Windows servers patching with AWS EC2 Systems Manager

Amazon EC2 Systems Manager is a collection of capabilities that helps you automate management tasks such as collecting system inventory, applying operating system patches, automating the creation of Amazon Machine Images (AMIs), and configuring operating systems and applications at scale. It is available at no cost to manage both your EC2 and on-premises resources!

Amazon EC2 Systems Manager relies on the Amazon Simple Systems Management Service (SSM) agent being installed on the guests. The SSM agent is pre-installed on Windows Server 2016 instances or Windows Server 2003-2012 R2 instances created from AMI’s published after November 2016. You need at least SSM agent version 2.0.599.0 installed on the target EC2 instance.

In this article we will focus on using Systems Manager to apply Windows Updates to EC2 instances. Patch Management is always an operational pain point so its welcome that AWS offers a solution.

You start by creating groups of instances by applying a tag called ‘Patch Group’. Then you create a group of patches by forming a patch baseline containing and excluding the patches you require (or use the AWS default patch baseline). At last you create a maintenance window to have your patch baseline attached and applied to a patch group. The actual ‘Patch Now’ run-command is nothing more than an API call, so there’s no obligation to use Maintenance Windows. Personally I’m a fan of Rundeck, so I’ll show you how to have the patches applied to the instances using both methods.

Configure your instances

The guest SSM agent setting inside with Windows OS requires permissions to connect to AWS EC2 Systems Manager. We grant these rights by creating an EC2 Service Role with the policy document ‘AmazonEC2RoleforSSM’ attached. Then you can attach this role to your instances. The instance also needs outbound internet connection to be able to connect to SSM. This can be either through an Internet Gateway or a NAT Gateway (or NAT Instance).

If you have this done right, your instance(s) should pop-up under ‘Managed Instances’ in the EC2 console:

Take note of the SSM Agent Version. As said earlier it must be at least version 2.0.599.0. The Systems Manager Service also requires a “Patch Group”-tag on the EC2 instance. The key for a patch group tag must be Patch Group. Note that the key is case sensitive. The value can be anything you want to specify, but the key must be Patch Group.

If done correctly, your tag will be picked up by SSM. You can confirm this on the ‘Managed Instances’ page:

Patch Baselines

AWS provides a default Patch Baseline called ‘AWS-DefaultPatchBaseline’. It auto-approves all critical and security updates with a ‘critical’ or ‘important’ classification seven days after they have been released by Microsoft. If you’re happy with that you can use this baseline. If you’re not, you can simply create your own according to your requirements: set approval for specific products and patch classifications, exclude a specific KB etc

Once your happy with your baseline, you can hit ‘Create’. Now assign it to one or more Patch Groups (or make it the default baseline and throw away the AWS one). Hit the ‘actions’ menu and chose ‘Modify Patch Groups’

Type the names of the Patch Groups you defined when tagging your instances

Your baseline is now attached to the specified patch groups. You can now start evaluating your instances against the baseline, and update them accordingly.

Patching

Applying the patch baseline to a specific instance or to a patch group is nothing more than executing an AWS SSM run command. You can schedule this run command through AWS SSM ‘Maintenance Windows’, a cron job on a server (like Rundeck) or manual through the AWS Console.

Let’s first check everything manually. In the AWS EC2 console, go to ‘Run Commands’ and create a new Run Command. Select the ‘AWS-ApplyPatchBaseline’ command document and pick an instance run this on. For the ‘operation’, choose ‘Scan’. This will evaluate the instance against the baseline without installing anything yet.

Once the run command finishes, you can go back to the ‘Managed Instances’ page. Highlight the instance(s) on which the run command was executed and click on the ‘Patch’ tab. Here you can see the result of the scan:

To actually install the missing updates, execute the same run command document, but now with the ‘Install’ operation. This will install the missing KBs to the instances and reboot them if needed.

Or execute the following aws cli command to accomplish the same:

aws ssm send-command --targets "Key=tag:Patch Group,Values=<PatchGroupName>" --document-name "AWS-ApplyPatchBaseline" --comment "Install|Check Windows Updates" --parameters Operation="<Install|Scan>"

Maintenance Windows

In stead of manually starting a run command or cron job, we can also use the AWS provided Maintenance Windows feature. Systems Manager Maintenance Windows let you define a schedule for when to perform actions on your instances such as patching the operating system. Each Maintenance Window has a schedule, a duration, a set of registered targets, and a set of registered tasks.

Before actually creating a Maintenance Window, we must configure a Maintenance Window role. We need this so Systems Manager can execute tasks in Maintenance Windows on our behalf. So we go to the IAM page and create a new role. We pick an “EC2 service role” type and make sure to attach the “AmazonSSMMaintenanceWindowRole” policy to it. Once the role is created, we must modify it. Click “edit Trust Relationships”. Add a comma after “ec2.amazonaws.com”, and then add “Service”: “ssm.amazonaws.com” to the existing policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com",
        "Service": "ssm.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Back to SSM now to actually create the Maintenance Window. Give it a useful name and specify your preferred schedule. I’m setting ‘every 30 minutes’ just for demonstration purposes, but in a real setup you would most probably choose something like ‘Every  Sunday’. You can also configure your own Cron expression.

This leaves us now with an empty Maintenance Window: there are no tasks nor targets associated yet.

To assign targets to the Maintenance Window, click on the “Register new targets” button on the “Targets” tab. We dynamically select the targets by using the “Patch Group” tag.

We will now have an ID linked to our “dev” Patch Group. This “Window Target ID” is used in the next step.

From the “tasks” tab of the Maintenance Window, click on “Schedule new task”. Pick the “AWS-ApplyPatchBaseline” document. Under “Registered Targets”, select the correct Window Target ID. For the operation, select “Install”. For the “Role”, select the IAM role with the AmazonSSMMaintenanceWindowRole attached to it (the one we created earlier). Set your preferred concurrency level and register the task by clicking on the blue button. The end result should look like this:

Now we have to wait for the schedule of the Maintenance Window. In this example we specified ‘every 30 minutes’ as a schedule, so the waiting shouldn’t take too long. Under the ‘History’ tab of the Maintenance Window you can follow all actions. The Maintenance Window will simply launch a Run Command, so you could go to that console screen too. If you enabled logging to S3, you could find the output of the Run Command over there. If not, you can view a (truncated) output via the Run Command itself:

Patch Summary for i-07ca5621af38f256d
PatchGroup          : dev
BaselineId          : pb-06101e06cf8506be6
SnapshotId          : 317f2b72-2612-4740-95af-c7b3d8fb6d1e
OwnerInformation    : 
OperationType       : Install
OperationStartTime  : 2017-05-29T11:00:14.0000000Z
OperationEndTime    : 2017-05-29T11:03:18.7164313Z
InstalledCount      : 1
InstalledOtherCount : 6
FailedCount         : 0
MissingCount        : 0
NotApplicableCount  : 3

EC2AMAZ-EA5SH8I - PatchBaselineOperations Installation Results - 2017-05-29T11:03:19.537

KbArticleId Installed   Message
----------- ----------- -----------
KB890830    Yes         Success

If we now go back to the “Managed Instances” page and look at the “Patch” tab of our test instance, we will see it is not missing any updates anymore!

Success! Another  on the Automation checklist!

Rutger