Server Maintenance – Snapshots & Maintenance Mode

Patching servers is an annoying, but critically necessary task. Every month, on every (virtual) server, we have to:

  1. Create a new snapshot (VMWare VCenter).
  2. Place the server in maintenance mode (SCOM).
  3. Download and install the OS and application updates.
  4. Reboot the server.
  5. Check that everything is working again.
  6. Delete the snapshot
  7. Take the server out of maintenance mode.

Given that we have several hundred servers (and growing), this process is taking an increasing amount of time each month. Over the years we’ve implemented various automated patching systems (WSUS, IBM BigFix, etc.) and they’ve worked reasonably well for managing the Download & Install step. The pain point lately has become the first two steps (snapshots and maintenance mode). Both processes are simple to complete using the VCenter web-based user interface and SCOM console. The problem is the volume of button clicks it takes to complete the process for ALL of the servers. Using the standard (web) user interfaces, over an hour of the monthly maintenance window can be lost to just getting the snapshots and maintenance mode tasks completed. Extrapolate that out over a year and we’re looking at over 1.5 DAYS of work-time lost to getting the servers ready to START applying updates. That’s not a statistic we want to publish to senior management. So, how to fix (or minimize) the problem? The answer to which is: Script It.

Let’s take a look at how to use PowerShell to automate the snapshot and maintenance mode tasks.

VMware Snapshots:
PowerShell integration with VMWare (VCenter, ESX hosts, etc) is made possible through the PowerCLI tool set that is available on the PSGallery.Find-Module VMware.PowerCLI

Once the PowerShell module is imported using: Import-Module VMware.VimAutomation.Core, there are 3 required pieces of information.

  • The VCenter server name.
  • The name of the virtual machine in VCenter to take a snapshot of.
  • A name or description for the snapshot.

A connection to the VCenter server is established using the Connect-VIServer command.

Connect-VIServer -Server "VCenter server name" -Protocol https;

Once the connection has been established, a new snapshot is created using the New-Snapshot command.
Note: connections to multiple VCenter servers can be established during a single PowerShell session, so each snapshot command should include the name of the hosting VCenter server, along with the VM name and snapshot name/description.

New-Snapshot -VM "virtual server name" -Name "snapshot name or description" -Server "VCenter server name";

After the snapshot has been created, disconnect from the VCenter server or the session will remain on the VCenter server.

Disconnect-VIServer -Server "VCenter server name";

Now that the snapshot has been created, let’s look at how to put the server in Maintenance Mode with SCOM.

System Center Operations Manager 2016 (SCOM) Maintenance Mode:

Unfortunately, the SCOM OperationsManager module for PowerShell is not currently available on the PSGallery, or as a stand-alone install. The only (official) way to get the PowerShell module is to install the SCOM admin console. Note: A few people have posted directions online for installing the SCOM module on a computer without the console. However, they appear to be a bit more hackish than I care to run on my computers/servers.

To set a server into maintenance mode, we’ll need a few pieces of information:

  • The name of the server hosting SCOM.
  • The name (FQDN) of the server to put into maintenance mode.
  • How long (minutes) to put the server into maintenance mode.
  • The reason for putting the server in maintenance mode (for the action’s comment field).

With these details in hand, we’ll start by importing the OperationsManager module. Then we can use the New-SCOMManagementGroupConnection, Get-SCOMMaintenanceMode and Start-SCOMMaintenanceMode commands to: connect to a SCOM management group, get the current maintenance mode status, and put the server into maintenance mode.

Import-Module OperationsManager;
New-SCOMManagementGroupConnection-ComputerName "SCOM Server Name";
Get-SCOMMaintenanceMode -Instance "Server Instance";
Start-SCOMMaintenanceMode -Instance "Server Instance" -EndTime <End Date/Time> -Comment "description" -Reason "PlannedApplicationMaintenance";

Now that we can take a snapshot and place the server into maintenance mode, there’s one issue left to resolve: The requirement of installing PowerCLI and the SCOM console on EVERY computer where the script might be run from. This script could reasonably be run from most desktops/laptops in the IT department, or from servers in the machine/server room, or from the IT virtual desktops (VMware Horizon VDI). That’s a lot of places to keep up to date with patches/upgrades for the PowerCLI/OperationsManager modules. Why not run the PowerShell commands remotely on servers that already have the modules installed? PowerShell has a PSRemoting commandlet. Let’s take a look at incorporating PSRemoting into our script.

PowerShell Remote Execution (PSRemoting):

To establish a remote PowerShell connection to a server with the PowerCLI or OperationsManager modules installed we’ll use New-PSSession and Invoke-Command.

$session = New-PSSession -ComputerName "Server Name";
Invoke-Command -Session $session -ScriptBlock $Blockcmd;

We’ll also want to be able to pass parameters from the local computer to the SCOM/PowerCLI servers. In PowerShell (v3 and newer) we can do this with the Using scope modifier for local variables. The Using scope syntax for each local variable (ie $ServerName) in the script block is $using:VariableName (ie $using:ServerName). So our SCOM script block ends up looking like:

$SCOMcmd = {Import-Module OperationsManager;
New-SCOMManagementGroupConnection -ComputerName "$using:SCOMServerName";
$MMState = Get-SCOMMaintenanceMode -Instance (Get-SCOMClassInstance -Name "$using:ServerFQDN");
Return $MMState;
}

Once we merge the PSRemoting functionality and Using scope modifier with our PowerCLI and SCOM commands, the final script looks like:

<#
.SYNOPSIS 
Creates a snapshot (VCenter) and places the server in Maintenance Mode (SCOM). 

.PARAMETER $VCenterName 
Name of the Virtual Center server that is host to the VM. 
If Null, the VCenter name is looked-up in the WMSysInfo database. 

.PARAMETER $VMName 
VM Name in VCenter 

.PARAMETER $SnapshotName 
Optional name for the Snapshot and Maintenance Mode comment. 

.PARAMETER $MaintenanceTimeInMinutes 
Number of minutes until SCOM maintenance mode will end. 

.PARAMETER $PowerCLIServer 
Name of the server with PowerCLI installed. 

.PARAMETER $SCOMServerName 
Name of the server with the SCOM console installed.
#>
Param (
[string]$VCenterName,
[string]$VMName,
[string]$ServerFQDN,
[string]$SnapshotName = $null,
[string]$MaintenanceTimeInMinutes = "90",
[string]$PowerCLIServer,
[string]$SCOMServerName
)

# Set default snapshot name if not provided
If (!$SnapshotName) {
    $SnapshotName = "$env:username - Server Maintenance (script)";
    }

# Create remote session to PowerCLI server
$VCsession = New-PSSession -ComputerName "$PowerCLIServer";

# Use Exit in block to force connection to close after snapshot. Else connection sometimes hangs.
$VCblock = {Import-Module VMware.VimAutomation.Core -DisableNameChecking;
    Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false;
    Connect-VIServer -Server $using:VCenterName -Protocol https;
    $Snapshot = New-Snapshot -VM $using:VMName -Name $using:SnapshotName -Server $using:VCenterName -Confirm:$false;
    Exit;
    }
Invoke-Command -Session $VCsession -ScriptBlock $VCblock;

# Get current number of snapshots on server
$VCsession = New-PSSession -ComputerName "$PowerCLIServer";
$VCblock = {Import-Module VMware.VimAutomation.Core -DisableNameChecking;
    Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false;
    Connect-VIServer -Server $using:VCenterName -Protocol https;
    $Count = (Get-Snapshot -VM $using:VMName).Count;
    Disconnect-VIServer -Server $using:VCenterName -Confirm:$false;
    Return $Count;
    }
# Snapshot Count is returned in the third array position.
$SnapshotCount = (Invoke-Command -Session $VCsession -ScriptBlock $VCblock)[2];

Remove-PSSession $VCsession;

# Set Maintenance Mode Time
$MaintenanceTime = ((Get-Date).AddMinutes($MaintenanceTimeInMinutes));

# Create remote session to SCOM server
$SCOMsession = New-PSSession -ComputerName "$SCOMServerName";

# Get Current Maintenance Mode State
$SCOMcmd = {Import-Module OperationsManager; 
    New-SCOMManagementGroupConnection -ComputerName "$using:SCOMServerName";
    $MMState = Get-SCOMMaintenanceMode -Instance (Get-SCOMClassInstance -Name "$using:ServerFQDN")
    Return $MMState;
    }
$MMState = Invoke-Command -Session $SCOMsession -ScriptBlock $SCOMcmd;  

# If not in Maintenance Mode, start Maintenance Mode
If (!$MMState) {
    $SCOMcmd = {Import-Module OperationsManager; 
    New-SCOMManagementGroupConnection -ComputerName "$using:SCOMServerName";
    Start-SCOMMaintenanceMode -Instance (Get-SCOMClassInstance -Name "$using:ServerFQDN") -EndTime $using:MaintenanceTime -Comment "$using:SnapshotName" -Reason "PlannedApplicationMaintenance"; 
    }
    Invoke-Command -Session $SCOMsession -ScriptBlock $SCOMcmd;
    $SCOMcmd = {Import-Module OperationsManager; 
        New-SCOMManagementGroupConnection -ComputerName "$using:SCOMServerName";
        $MMState = Get-SCOMMaintenanceMode -Instance (Get-SCOMClassInstance -Name "$using:ServerFQDN")
        Return $MMState;
        }
    $MMState = Invoke-Command -Session $SCOMsession -ScriptBlock $SCOMcmd;
    }

# Close remote session
Remove-PSSession $SCOMsession;

$Endtime = ($MMState).ScheduledEndTime;

# Convert UTC to local
$CurrentTimeZone = (Get-WmiObject win32_timezone).StandardName
$TZ = [System.TimeZoneInfo]::FindSystemTimeZoneById($CurrentTimeZone)
$LocalTime = [System.TimeZoneInfo]::ConvertTimeFromUtc($Endtime, $TZ)

$Status = @"
**********************************
Snapshot count for $VMName`: $SnapshotCount
Maintenance Mode will end at: $Localtime
**********************************
"@

Write-Output $Status;

1 thought on “Server Maintenance – Snapshots & Maintenance Mode”

Leave a Reply

Your email address will not be published. Required fields are marked *