We were recently working on an AWS setup which involved a Network LoadBalancer (NLB) with a TCP listener and a requirement for sticky sessions. As we were seeing some strange behavior which we couldn’t immediately explain and which might be linked to the session stickiness we decided to make a small test setup.

The problem

Unlike an ALB where session stickiness is accomplished with cookies, the NLB uses a built-in 5-tuple hash table in order to maintain stickiness across backend servers. We access the NLB through its DNS name, which actually returns the IPs of the two NLB endpoints in a round-robin fashion with a TTL of 60 seconds.

We were looking for an answer on the following question: if our end-user would resolve the DNS and pick the IP of the first NLB endpoint to start the connection, the session will be routed towards one of the backend servers. But after 60 seconds the client could potentially re-issue the DNS query and start its connection with the other NLB endpoint. How will the stickiness and cross-zone loadbalancing behave? Will our end-user connection be routed to the initial server again, even if that means crossing AZ boundaries?

The situation

We started with a classical setup comprising of an NLB with an endpoint in each AZ and a Targetgroup having one instance as target in each AZ.

Scenario #1

• Cross-Zone loadbalancing: Disabled
• TargetGroup Stickiness: Disabled

How does it behave?

1. Client connects to the IP of the first NLB node: the connection is redirected to the server in AZ 1.
2. Client connects to the IP of the second NLB node: the connection is redirect to the server in AZ 2.

Since there is only one healthy target per AZ and cross-zone loadbalancing is not enabled, this situation results in ‘AZ-stickiness’: the traffic remains in the AZ in which it arrived. The setup relies on DNS to distribute client connections evenly across both NLB endpoints, but there is nothing to guarantee that a specific user connection is always directed to the same NLB endpoint, let alone to the same backend server.

Scenario #2

• Cross-Zone loadbalancing: Enabled
• TargetGroup Stickiness: Disabled

Allowing cross-zone loadbalancing and not requiring any stickness. This should give us complete randomness, shouldn’t it?

And so it does. We’re now completely randomized and hit every backend server, irrespective of our NLB endpoint ‘point of entry’. Works as expected.

Scenario #3

• Cross-Zone loadbalancing: Disabled
• TargetGroup Stickiness: Enabled

We’ve enabled stickiness on the targetgroup now, and disabled the cross-zone loadbalancing again. Let’s hope our client connection is now sticky to a specific backend server.

1. Client connects to the IP of the first NLB node: the connection is redirected to the server in AZ 1.
2. Client connects to the IP of the second NLB node: the connection is redirect to the server in AZ 2.

Ok wait, we’ve asked our TargetGroup to be sticky, but still our connection is balanced over both backend servers? What’s going on?

The fact that our NLB is not allowing cross-zone loadbalancing seems to prevent the connection from reaching the same backend every time. The connection enters via NLB endpoint 1 but stickiness has decided that the connection should go to server in AZ 2? Stickiness fails, the disabled cross-zone loadbalancing wins…

With only one healthy backend per AZ this behaves the same as not enabling stickiness at all. We’re pretty sure that with more than one backend per AZ the stickiness is maintained…within that AZ only. Interesting!

Scenario #4

• Cross-Zone loadbalancing: Enabled
• TargetGroup Stickiness: Enabled

Let’s solve this. We’ve enabled both cross-zone loadbalancing and targetgroup stickiness. We should hit the same backend server every time now.

And so it does. Only now we reach true stickiness and hit the same backend server every time, no matter how hard we try by entering via the loadbalancer node in the other AZ.

The conclusion

If you don’t allow cross-zone loadbalancing, then stickiness is only active within AZ boundaries. As DNS round-robin could direct a client to a different point of entry after the TTL has expired, strict stickiness is not guaranteed.

So if you really need stickiness to a specific backend target, you need to allow cross-zone loadbalancing (and live with the extra cost of inter-AZ traffic). Only now do the different loadbalancer nodes share the hash table of “client-to-target” stickiness.

Kinda logic, though…

PS: NLB idle timeout for TCP connections is 350 seconds. Once the timeout is reached or the session is terminated, the NLB will forget the stickiness and incoming packets will be considered as a new flow and could be loadbalanced to a new target.

The post Why AWS NLB stickiness is not always sticky appeared first on Cloudar.

The post 10 random things you probably didn’t know about Cloudar appeared first on Cloudar.

We will use a custom resource written in Python that will be able to create ACM certificates with DNS validation. The custom resource will also automatically validate this certificate if the validation domain is managed by a Route53 hosted zone. We will also be able to specify an AWS region to create the certificate in, this region is independent of the Cloudformation stack region which for example makes it possible to deploy a certificate in region us-east-1 (to use with cloudfront) while deploying the stack in region eu-west-1. The resource will also provide the certificate arn as an output parameter so it can be used by other resources in the stack. Lastly when you delete the custom resource it will cleanup all validation records and the certificate itself.

Requirements:

• Python3
• Pip
• Bash
• Zip
• An S3 bucket to deploy the custom resource package on
• A hosted zone for the validation record

Implementation:

Let’s get started by downloading all the required code from our GitHub repository.

Step1: Uploading the custom resource package

In this step we are going to prepare the custom resource package and upload it to an S3 bucket.

First we go into the custom resource directory.
cd cloudar-acm-plus-custom-resource

Next we execute a script to install all required dependencies.
sh install_dependencies

Now we are ready to create the package.
sh pack_custom_resource

You will now find the zipfile ‘cloudar-acm-plus-custom-resource.zip’ in ‘cloudar-acm-plus-custom-resource/packed’, upload this zip file to your S3 bucket.

Step2: Creating a Cloudformation template

Now we can create a Cloudformation template in which we use this custom resource to create an ACM certificate.
You can use the template ‘cfn.yaml’ as an example.

First create a Lambda resource as following

Use the name of your bucket for the property ‘S3Bucket’ .

Next we create the custom resource.

We can set the following properties here:

• DomainName: (REQUIRED type:String) The domain name for the acm certificate.
• AdditionalDomains: (OPTIONAL type:List) Additional domains for the acm certificate
• ValidationDomain: (REQUIRED type:string) The domain name for the validation domain of the acm certificate
• HostedZoneId: (REQUIRED type:string) The hosted zone id for the validation domain of the acm certificate
• CertificateRegion: (REQUIRED type:string) The region to deploy the acm certificate in
• IdempotencyToken: (REQUIRED type:string pattern: \w+) The idempotency token for the create call of the acm certificate doc: https://docs.aws.amazon.com/acm/latest/APIReference/API_RequestCertificate.html#ACM-RequestCertificate-request-IdempotencyToken
• CertificateTags: (OPTIONAL type:list) The tags for the acm certificate

In order for the DNS record cleanup and delete certificate functionality to work when you delete the Cloudformation stack it is important to set the following output.

As you can see we can access the arn of the certifcate created by the custom resource with the GetAtt function on the resource.
!GetAtt CreateCertificateCustomResource.certificate_arn

Step3: Deploy the cloudformation

Finally the only thing left to do is deploy the Cloudformation template.
Once the deploy is started Cloudformation will create the Lambda containing the code from step1 and start a custom resource which will create the certificate and validation records. Once the status of the certificate becomes ‘ISSUED’ the custom resource will finish successfully and report the arn of the certificate back to Cloudformation. We can now further use this arn in other resources in the Cloudformation template.
When you delete the Cloudformation stack, the custom resource will cleanup the validation records in the hosted zone and delete the certificate.

CREATE_COMPLETE

The post Validate ACM certificates in Cloudformation appeared first on Cloudar.

You just finished setting up your super-duper AWS environment..

Highly available & Fault Tolerant: check!
Backups in place: check!
MFA enforced: check!
Security Groups and NACLs: check!
CloudTrail enabled: check!

You even deserve bonus points for activating Amazon GuardDuty and putting AWS WAF & Shield in front of your CloudFront distribution and loadbalancers.
Time to lean back with a bit of smugness, while you take a sip of your well-deserved cup of coffee.

Seems like you have covered all your bases, or have you?

One often overlooked topic is security incident response.
While a lot of security incidents can (and should) be mitigated through automation,
some incidents will require manual intervention, such as information and evidence gathering during and after a successful malicious attack on one of your instances.

Effective incident response and forensics require preparation, well ahead of time.
It is critical to have your forensics and remediation tools readily available for whenever the proverbial shit hits the fan.
Documenting your investigative steps and being able to execute them swiftly, will contribute to a well-controlled, thorough and effective investigation.

In this blog post, I’d like to focus on some of the first steps you might take in your investigation:

• Building a forensics workstation and  taking a memory dump of a compromised instance.
• Preparation steps and tools, both for windows and linux.
• The forensics investigation process.
• An investigation of a real memory dump.

Although some great tools are already available from Threadresponse (such as AWS_IR and Margarita Shotgun), we will build a solution ourselves, in order to gain a thorough knowledge on how the process works behind the scenes.
Creating a memory dump is  something that  should be done immediately, as it provides a snapshot of the memory at the time of the attack.
The dump can then be analyzed and used to build a timeline, improve security after the fact and optionally provide evidence for any follow up with law enforcement.

Tools

• Linux: LiME ( Linux Memory Extractor )
• Windows: FTK or Belkasoft Live  RamCapturer
  ( both are free, but require registration, you get the download link in a mail )
• Volatility

Preparation Steps

Create Quarantine and Forensic Security Groups

1. Forensics Security Group.
   This SG will be attached to your Forensics Workstation later on.

   aws ec2 create-security-group --group-name ForensicsSG \
   --description "Forensics SG" --vpc-id <your vpc id> \
   --profile <your profile>

   This will output something like this:

   {
       "GroupId": "sg-22222222222222222"
   }

   Now we use this GroupId to add an inbound rule that allows  SSH  connection towards our Forensics Workstation

   aws ec2 authorize-security-group-ingress --group-id sg-22222222222222222 \
   --protocol tcp --port 22 --cidr <your cidr> --profile <your profile

   By default , outbound everything is allowed, and in this case, that is ok, so we leave it at that.

2. Quarantine Security Group
   This SG will be attached to the compromised instance, to isolate it from any network, except the forensics network.

   aws ec2 create-security-group --group-name QuarantineSG \
   --description "Quarantine SG" --vpc-id <your vpc id> \
   --profile <your profile>
   

   Output will look similar to the following:

   {
       "GroupId": "sg-111111111111111111"
   }

   Remove all rules from outbound ( egress )

   aws ec2 revoke-security-group-egress --group-id sg-11111111111111111 \
   --ip-permissions '[{"IpProtocol": "-1","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"Ipv6Ranges": [{"CidrIpv6": "::/0"}]}]' \
   --profile <your profile>

   Add rules to allow access from the Forensics Security Group

   aws ec2 authorize-security-group-ingress --group-id sg-11111111111111111 \
   --ip-permissions '[
       {"IpProtocol":"tcp","FromPort":22,"ToPort":22,"UserIdGroupPairs":[
           {"GroupId": "sg-22222222222222222","Description": "SSH access from the ForensicsSG"}
       ]},
       {"IpProtocol":"tcp","FromPort":4444,"ToPort":4444,"UserIdGroupPairs":[
           {"GroupId":"sg-22222222222222222","Description":"Access from the ForensicsSG for LiME dump over TCP"}
       ]},
       {"IpProtocol":"tcp","FromPort":3389,"ToPort":3389,"UserIdGroupPairs":[
           {"GroupId":"sg-22222222222222222","Description":"RDP access from the ForensicsSG"}
       ]}
       ]' \
   --profile <your profile>

3. Create Isolation Functionality
   Create lambda-execution-forensics-trust-policy.json file with following content:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "lambda.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }

   Create the Lambda Execution role, forensics-lambda-exec-role, using the policy file you just created:

   aws iam create-role \
   --role-name forensics-lambda-exec-role  \
   --assume-role-policy-document file://lambda-execution-forensics-trust-policy.json \
   --profile <your profile>

   The output should look like this:

   {
       "Role": {
           "AssumeRolePolicyDocument": {
               "Version": "2012-10-17", 
               "Statement": [
                   {
                       "Action": "sts:AssumeRole", 
                       "Effect": "Allow", 
                       "Principal": {
                           "Service": "lambda.amazonaws.com"
                       }
                   }
               ]
           }, 
           "RoleId": "AROAILWIMFHDIIGTDST7I", 
           "CreateDate": "2018-11-20T20:42:44Z", 
           "RoleName": "forensics-lambda-exec-role", 
           "Path": "/", 
           "Arn": "arn:aws:iam::111111111111:role/forensics-lambda-exec-role"
       }
   }

   Create lambda-forensics-policy.json with following content:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Action": [
                   "logs:CreateLogStream",
                   "logs:PutLogEvents",
                   "logs:CreateLogGroup"
               ],
               "Resource": "arn:aws:logs:*:*:*",
               "Effect": "Allow"
           },
           {
               "Action": [
                   "ec2:Describe*",
                   "ec2:ModifyNetworkInterfaceAttribute",
                   "ec2:ModifyInstanceAttribute"
               ],
               "Resource": "*",
               "Effect": "Allow"
           }
       ]
   }

   Create the policy:

   aws iam create-policy --policy-name lambda-execute-forensics-policy \
   --policy-document file://lambda-forensics-policy.json 
   --profile <your profile>

   This will output something like this:

   {
       "Policy": {
           "PolicyName": "lambda-execute-forensics-policy", 
           "PermissionsBoundaryUsageCount": 0, 
           "CreateDate": "2018-11-20T20:56:04Z", 
           "AttachmentCount": 0, 
           "IsAttachable": true, 
           "PolicyId": "ANPAIG23L72JCRWAQQEFE", 
           "DefaultVersionId": "v1", 
           "Path": "/", 
           "Arn": "arn:aws:iam::111111111111:policy/lambda-execute-forensics-policy", 
           "UpdateDate": "2018-11-20T20:56:04Z"
       }
   }

   Attach the policy to the forensics-lambda-exec-role, using the ARN found in the output from the previous comand

   aws iam attach-role-policy --role-name forensics-lambda-exec-role \
   --policy-arn arn:aws:iam::111111111111:policy/lambda-execute-forensics-policy \
   --profile <your profile>

   Create index.py with following content:

   import json
   import boto3
   import os
   
   ec2 = boto3.resource('ec2')
   quarantine_sg = os.environ['QUARANTINE_SG']
   
   def set_SecurityGroup(instance):
     oldsecuritygroups = {}
     interfaces = instance.network_interfaces
     for interface in interfaces:
       oldsecuritygroups[interface.id] =  interface.groups
       interface.modify_attribute(Groups = [quarantine_sg])
   
     return oldsecuritygroups
   
   def lambda_handler(event, context):
     instance = ec2.Instance(event['instance_id'])
     # Remove current Security Groups
     orig_groups = set_SecurityGroup(instance)
     return {
       'statusCode': 200,
       'body': "OK",
       'replaced_sgs': orig_groups
     }
   
   

   This function will replace all Security Groups on all attached network interfaces of an instance with the Quarantine Security Group we created earlier. When you discover that an instance is compromised, you can invoke this function, and it will isolate the instance.  It fetches the Security Group from the environment, and takes the instance id of the compromised instance as input as json in the following format:

   {
       "instance_id": "i-xxxxxxxxxxxxxxxxx"
   }

   The function will output the removed Security Groups so you have a reference for later on.

   zip the file to index.zip

   zip index.zip index.py

   Create the Lambda function, using the IAM Policy ARN from the output of the role creation (forensics-lambda-exec-role)

   aws lambda create-function --function-name forensics-isolate-instance \
   --zip-file fileb://index.zip \
   --role arn:aws:iam::111111111111:role/forensics-lambda-exec-role \
   --handler index.lambda_handler --runtime python3.6 \
   --environment Variables={QUARANTINE_SG=sg-11111111111111111} \
   --profile <your profile>

   OK, Done, now let s move on and create a forensics workstation.

Build a Forensics Workstation

1. Create EC2 Instance Role for the forensics Workstation
   Create ec2-forensics-trust-policy.json file with following content:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "ec2.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }

   Create the EC2 Instance role, ec2-forensics-role, using the Policy file you just created:

   aws iam create-role --role-name ec2-forensics-role \
   --assume-role-policy-document file://ec2-forensics-trust-policy.json \
   --profile <your profile>

   Output:

   {
       "Role": {
           "AssumeRolePolicyDocument": {
               "Version": "2012-10-17", 
               "Statement": [
                   {
                       "Action": "sts:AssumeRole", 
                       "Effect": "Allow", 
                       "Principal": {
                           "Service": "ec2.amazonaws.com"
                       }
                   }
               ]
           }, 
           "RoleId": "AROAIGXSH7RCMPCVPY4QA", 
           "CreateDate": "2018-11-21T10:34:07Z", 
           "RoleName": "ec2-forensics-role", 
           "Path": "/", 
           "Arn": "arn:aws:iam::111111111111:role/ec2-forensics-role"
       }
   }
   

   Create ec2-forensics-policy.json with following content:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowEC2InstanceEC2Forensics",
               "Effect": "Allow",
               "Action": [
                   "ec2:Describe*",
                   "ec2:AuthorizeSecurityGroupIngress",
                   "ec2:ModifyVolumeAttribute",
                   "ec2:CreateKeyPair",
                   "ec2:ReportInstanceStatus",
                   "ec2:ModifySnapshotAttribute",
                   "ec2:RevokeSecurityGroupEgress",
                   "ec2:ImportKeyPair",
                   "ec2:CreateTags",
                   "ec2:StopInstances",
                   "ec2:RevokeSecurityGroupIngress",
                   "ec2:AttachVolume",
                   "ec2:ImportVolume",
                   "ec2:ModifySubnetAttribute",
                   "ec2:CreateSnapshot",
                   "ec2:RebootInstances",
                   "ec2:ImportInstance",
                   "ec2:ResetSnapshotAttribute",
                   "ec2:ImportSnapshot",
                   "ec2:CopySnapshot",
                   "ec2:CreateImage",
                   "ec2:CopyImage",
                   "ec2:GetLaunchTemplateData",
                   "ec2:ImportImage",
                   "ec2:DetachVolume",
                   "ec2:CreateFlowLogs",
                   "ec2:GetConsoleOutput",
                   "ec2:CreateSecurityGroup",
                   "ec2:CreateNetworkAcl",
                   "ec2:ModifyInstanceAttribute",
                   "ec2:AuthorizeSecurityGroupEgress",
                   "ec2:DetachNetworkInterface",
                   "ec2:CreateNetworkAclEntry"
               ],
               "Resource": "*"
           },
           {
               "Sid": "AllowEC2InstanceInvokeLambda",
               "Effect": "Allow",
               "Action": "lambda:InvokeFunction",
               "Resource": "arn:aws:lambda:eu-west-1:111111111111:function:forensics-isolate-instance"
           }
       ]
   }

   Create the ec2-forensics-policy IAM Policy:

   aws iam create-policy --policy-name ec2-forensics-policy \
   --policy-document file://ec2-forensics-policy.json \
   --profile <your profile>

   Attach the policy to the ec2-forensics-role, using the ARN found in the output from the previous command.
   The Forensics Workstation also needs access to S3 and AWS Systems Manager, so we also include some AWS Managed Policies

   aws iam attach-role-policy --role-name ec2-forensics-role \
   --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM \
   --profile <your profile>
   aws iam attach-role-policy --role-name ec2-forensics-role \
   --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess \
   --profile <your profile>
   aws iam attach-role-policy --role-name ec2-forensics-role \
   --policy-arn arn:aws:iam::111111111111:policy/ec2-forensics-policy \
   --profile <your profile>

   Create an Instance Profile, called ec2-forensics-profile

   aws iam create-instance-profile \
   --instance-profile-name ec2-forensics-profile \
   --profile <your profile>

   Attach the ec2-forensics-role to the ec2-forensics-profile

   aws iam add-role-to-instance-profile \
   --instance-profile-name ec2-forensics-profile \
   --role-name ec2-forensics-role \
   --profile <your profile>

2. Provision the Forensics Workstation
   Create user-data.txt script with following content:

   #!/bin/bash
   
   # Install prerequisites
   sudo yum -y update
   sudo yum install python-pip pcre-tools gcc autoconf automake libtool nc git kernel-devel libdwarf-tools
   pip install distorm3 pycrypto pillow openpyxl ujson pytz IPython
   
   # Install Volatility
   cd /home/ec2-user
   wget http://downloads.volatilityfoundation.org/releases/2.6/volatility-2.6.zip
   unzip volatility-2.6.zip
   mv volatility-master volatility
   chown -R ec2-user.ec2-user volatility
   
   # Install LiME
   git clone https://github.com/504ensicsLabs/LiME.git
   chown -R ec2-user.ec2-user LiME

   Create the EC2 Instance

   aws ec2 run-instances --image-id ami-09693313102a30b2c \
   --count 1 --instance-type t3.micro --key-name MyKeyPair \
   --security-group-ids sg-22222222222222222 \
   --subnet-id subnet-33333333333333333 \
   --user-data file://user-data.txt \
   --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value="My Forensics WS"}]' \
   --iam-instance-profile Name=ec2-forensics-profile \
   --profile <your profile>

   Tip: You can now fetch the latest Amazon AMI by querying the SSM Parameter Store:

   aws ssm get-parameters \
   --names /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 \
   --profile <your profile>
   {
       "InvalidParameters": [], 
       "Parameters": [
           {
               "Name": "/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2", 
               "LastModifiedDate": 1542668357.322, 
               "Value": "ami-09693313102a30b2c", 
               "Version": 11, 
               "Type": "String", 
               "ARN": "arn:aws:ssm:eu-west-1::parameter/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"
           }
       ]
   }
   

   We now have a basic workstation.

3. Create a forensic volume
   We will need a volume we can attach/detach from any instance to temporarily store memory dumps and provide tools to the compromised instance.
   Since we need to support both Windows and Linux, we create an exfat filesystem on it, as it is supported on all platforms.

   aws ec2 create-volume --availability-zone eu-west-1a \ 
   --size 100 --volume-type gp2 --tag-specifications \
   'ResourceType=volume,Tags=[{Key=Name,Value=exFat-Forensics-Volume}]' \
   --profile <your profile>

   This creates a 100 GB volume. You can lower the size , but make sure that it is comfortably bigger then the largest ram size on any of your instances.

   Connect to the Forensics Workstation and attach the new volume ( you probably need to run aws configure to set your default region first )

   aws ec2 attach-volume --device /dev/sdX \
   --volume-id <volumeid> \
   --instance-id $(curl http://169.254.169.254/latest/meta-data/instance-id)

   Partition and format the disk

   sudo fdisk /dev/sdh
   
   Welcome to fdisk (util-linux 2.30.2).
   Changes will remain in memory only, until you decide to write them.
   Be careful before using the write command.
   
   Device does not contain a recognized partition table.
   Created a new DOS disklabel with disk identifier 0x601015b4.
   
   Command (m for help): p
   Disk /dev/sdh: 100 GiB, 107374182400 bytes, 209715200 sectors
   Units: sectors of 1 * 512 = 512 bytes
   Sector size (logical/physical): 512 bytes / 512 bytes
   I/O size (minimum/optimal): 512 bytes / 512 bytes
   Disklabel type: dos
   Disk identifier: 0x601015b4
   
   Command (m for help): n
   Partition type
      p   primary (0 primary, 0 extended, 4 free)
      e   extended (container for logical partitions)
   Select (default p): p
   Partition number (1-4, default 1): 
   First sector (2048-209715199, default 2048): 
   Last sector, +sectors or +size{K,M,G,T,P} (2048-209715199, default 209715199): 
   
   Created a new partition 1 of type 'Linux' and of size 100 GiB.
   
   Command (m for help): t
   Selected partition 1          
   Hex code (type L to list all codes): 7
   Changed type of partition 'Linux' to 'HPFS/NTFS/exFAT'.
   
   Command (m for help): w
   The partition table has been altered.
   Calling ioctl() to re-read partition table.
   Syncing disks.
   
   

   sudo mkfs.vfat /dev/sdh1

   Mounting the volume

   sudo mount /dev/sdhX /mnt

   Copy your local LiME and Volatility folders to the volume, to be able to compile a kernel module or create a Volatility profile on the fly, just in case.
   For Windows, we copy  RamCapturer or/and FTK Imager to the volume as well.

   Once you have all tools on the volume, unmount it, detach it and make a snapshot.

   Next step, we will gather LiME kernel modules and create Volatility Profiles for instances, running in our environment.
   The LiME kernel module needs to be loaded on the compromised instance

Create LiME kernel modules and Volatility Profiles

Note that you need to compile the kernel module for the EXACT kernel version , running on your instances. If you don’t patch your systems and/or have a lot of different Linux flavors, you will have a hard time maintaining the lime kernel modules and volatility profiles.

You have 2 options in obtaining a lime kernel module matching your kernel:

1. threadresponse lime module repository
   https://threatresponse-lime-modules.s3.amazonaws.com/
   This opens an xmlfile in which the available lime modules can be found
   Here is an excerpt of that xml file.

   <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <Name>threatresponse-lime-modules</Name>
   <Prefix/>
   <Marker/>
   <-- SNIP -->
   <Contents>
   <Key>
   modules/lime-2.6.32-131.0.15.el6.centos.plus.x86_64.ko
   </Key>
   <LastModified>2018-09-06T18:40:37.000Z</LastModified>
   <ETag>"93efeac2519a4c6a573601d203416098"</ETag>
   <Size>1098692</Size>
   <StorageClass>STANDARD</StorageClass>
   </Contents>
   <Contents>
   <Key>
   modules/lime-2.6.32-131.0.15.el6.centos.plus.x86_64.ko.sig
   </Key>
   <LastModified>2018-09-06T18:41:34.000Z</LastModified>
   <ETag>"c4d44af3b2265e55e8c23b8dc62d8828"</ETag>
   <Size>566</Size>
   <StorageClass>STANDARD</StorageClass>
   </Contents>
   ..

   The reference to the kernel modules  ( in the above example on line 8 ) is  the s3 object key to the object.
   So if your kernel is 2.6.32-131.0.15.el6, you can download it from https://threatresponse-lime-modules.s3.amazonaws.com/modules/lime-2.6.32-131.0.15.el6.centos.plus.x86_64.ko

2. build your own repository
   If you cannot find the lime modules for kernels running in your environment, you can build your own.
   Install volatility and LiME on either an existing instance, or launch a new instance of  the Linux distribution for which you want to create a module and volatility profile.For Redhat/CentOS/ Amazon Linux, you can use the userdata from the Forensics Workstation (2 .Provision the Forensics Workstation)
   For Debian/Ubuntu you can use this user-data content:

   #!/bin/bash
   sudo apt -y update
   sudo apt -y install libelf-dev libdwarf-dev dwarfdump zip make gcc
   cd /home/ubuntu
   wget http://downloads.volatilityfoundation.org/releases/2.6/volatility-2.6.zip
   unzip volatility.zip
   mv volatility-master volatility
   chown -R ubuntu.ubuntu volatility
   # Install LiME
   git clone https://github.com/504ensicsLabs/LiME.git
   chown -R ubuntu.ubuntu LiME

   1. Compile the LiME module
      First, Make sure you have kernel headers, source and images installed on this instance.

      cd ~/LiME/src
      make

      This should have created a file called lime-$(uname -r).ko
      If you need to create a module for a different kernel version ( for example for an older unpatched instance ),
      install the required version and change the above command to:

      make KVER=<kernel version>

      Note that this does not require the targeted kernel version to be active

   2. Create Volatility Profiles

      cd ~/volatility/tools/linux
      
      # change os to reflect your distro
      os="ubuntu"
      # change kernel version to the kernel version you want to compile for
      # do make sure this kernel is installed ( including headers, source and boot image
      # however, the kernel version does not need to be actively running
      kernel_version=$(uname -r)
      make KVER=${kernel_version}
      zip ~/${os}-${kernel_version}.zip module.dwarf /boot/System.map-${kernel_version}

   3. Copy the lime module (.ko file) and volatility profile  (.zip file) to an S3 Bucket that acts as a central repository.

Incident Workflow

1. Logon onto the Forensics WorkStation
   ssh into your Forensics Workstation in 2 terminal windows. ( Use ssh-agent, ssh-add and ssh -A)
2. Isolate the compromised instance

   aws lambda invoke --function-name forensics-isolate-instance \
   --payload '{"instance_id": "<compromised instance id>"}' /tmp/output.txt
   

3. Create a snapshot of the compromised instance

   aws ec2 create-snapshot \
   --volume-id <volume-id of compromised instances root volume> \
   --tag-specifications \
   'ResourceType=snapshot,Tags=[{Key=Name,Value=compromised-instance-snap}]'

4. Attach the Forensics Volume to the compromised EC2 Instance and mount it.
   On The Forensics Workstation:

   aws ec2 attach-volume --device /dev/sdX \
   --volume-id <forensics volume id> \
   --instance-id <compromised instance id>

   On the compromised EC2 Instance:

   Linux:
   ssh into the compromised instance, fetch the kernel version and mount the volume.
   We need the kernel version for the next steps

   uname -r
   sudo mount /dev/sdX1 /mnt

   Windows:
   RDP into the compromised instance ( you need to setup an SSH Tunnel via the Forensics Workstation )
   Mount the volume via Computer Management -> Disk Management
   
   The new Volume is attached and identified as Disk 1 , but it is in an offline state.
   Right-click on Disk 1 and select ‘Online’
   
   The Volume is now online and mapped to D:
   

   On Windows, you can now skip to Step 7

5. In the other terminal ( Forensics WS ), fetch the lime kernel module from your S3 Bucket or the Threadresponse repository
6. scp the LiME module to the compromised instance
7. Run the memory dump on the compromised instance
   On Linux:

   sudo insmod /path/to/lime-$(uname -r).ko "path=/mnt/ram.lime format=lime digest=sha1"

   This will create the memory dump file ram.lime and the digest file ram.sha1 on the forensics volume.

   On Windows:
   Open File Explorer, and go to D:\
   If  RamCapturer is not yet unzipped, unzip RamCapturer.zip first.
   Then run D:\RamCapturer\x64\RamCapturer.exe as Administrator
   
   
   Save the dump to D:\ and run ‘Capture!’
   The dump will be saved as YYYYMMDD.mem where YYYYMMDD is the current date.

8. Fetch the memory dump onto the Forensics WorkStation
   unmount the Forensics Volume on Linux, or , on Windows, put it offline again using the Disk Management.
   Detach the volume from the compromised EC2 Instance and attach it back to the Forensics Workstation.
9. Stop the Compromised instance.

Sample Results from Memory Dumps

Volatility requires a profile matching your kernel. For Windows this is already included , but for Linux , you might need the volatility profile to be imported into your volatility setup.
let’s first test if the profile for your kernel is already configured:

cd volatility
python vol.py --info | grep Profile
Volatility Foundation Volatility Framework 2.6
Profiles
Linuxamzn-4_14_72-73_55_amzn2_x86_64x64 - A Profile for Linux amzn-4.14.72-73.55.amzn2.x86_64 x64
Linuxamzn-4_14_77-81_59_amzn2_x86_64x64 - A Profile for Linux amzn-4.14.77-81.59.amzn2.x86_64 x64
VistaSP0x64                             - A Profile for Windows Vista SP0 x64
VistaSP0x86                             - A Profile for Windows Vista SP0 x86
VistaSP1x64                             - A Profile for Windows Vista SP1 x64
VistaSP1x86                             - A Profile for Windows Vista SP1 x86
VistaSP2x64                             - A Profile for Windows Vista SP2 x64
VistaSP2x86                             - A Profile for Windows Vista SP2 x86
Win10x64                                - A Profile for Windows 10 x64
Win10x64_10240_17770                    - A Profile for Windows 10 x64 (10.0.10240.17770 / 2018-02-10)

...

If your kernel is not listed, you can add it by copying the volatility profile ( created in section 2.2 Create Volatility Profile )
to the volatility/plugins/overlays/linux/ directory.
Rerunning the above command should show your added profile.

OK, Pièce de résistance: some results:

Fetching lsof for the linux memory dump:

python vol.py -f /mnt/ram-20181122.lime \
--profile Linuxamzn-4_14_72-73_55_amzn2_x86_64x64 linux_lsof

Volatility Foundation Volatility Framework 2.6
Offset             Name                           Pid      FD       Path
------------------ ------------------------------ -------- -------- ----
0xffff88001c628000 systemd                               1        0 /dev/null
0xffff88001c628000 systemd                               1        1 /dev/null
0xffff88001c628000 systemd                               1        2 /dev/null
0xffff88001c628000 systemd                               1        3 anon_inode:[6744]
0xffff88001c628000 systemd                               1        4 anon_inode:[6744]
0xffff88001c628000 systemd                               1        5 anon_inode:[6744]
0xffff88001c628000 systemd                               1        6 /sys/fs/cgroup/systemd
0xffff88001c628000 systemd                               1        7 anon_inode:[6744]
0xffff88001c628000 systemd                               1        8 socket:[14109]
0xffff88001c628000 systemd                               1        9 /proc/1/mountinfo

-- SNIP --

0xffff8800182525c0 sudo                              23415        3 pipe:[1758814]
0xffff8800182525c0 sudo                              23415        4 pipe:[1758814]
0xffff8800182525c0 sudo                              23415        5 socket:[1758834]
0xffff8800182525c0 sudo                              23415        6 socket:[1758839]
0xffff880018254b80 insmod                            23416        0 /dev/pts/0
0xffff880018254b80 insmod                            23416        1 /dev/pts/0
0xffff880018254b80 insmod                            23416        2 /dev/pts/0
0xffff880018254b80 insmod                            23416        3 /mnt/lime-modules/amazon/lime-4.14.72-73.55.amzn2.x86_64.ko

List all established connections on Linux

python vol.py -f /mnt/ram-20181122.lime --profile Linuxamzn-4_14_72-73_55_amzn2_x86_64x64 linux_netstat | grep EST
Volatility Foundation Volatility Framework 2.6
TCP      10.100.4.6      :   22 10.100.4.111    :44854 ESTABLISHED                  sshd/23161
TCP      10.100.4.6      :   22 10.100.4.111    :44854 ESTABLISHED                  sshd/23179

List all open ports on Linux

python vol.py -f /mnt/ram-20181122.lime --profile Linuxamzn-4_14_72-73_55_amzn2_x86_64x64 linux_netstat | grep LISTEN
Volatility Foundation Volatility Framework 2.6
TCP      0.0.0.0         :  111 0.0.0.0         :    0 LISTEN                    rpcbind/2662 
TCP      ::              :  111 ::              :    0 LISTEN                    rpcbind/2662 
TCP      127.0.0.1       :   25 0.0.0.0         :    0 LISTEN                     master/3143 
TCP      0.0.0.0         :   22 0.0.0.0         :    0 LISTEN                       sshd/3273 
TCP      ::              :   22 ::              :    0 LISTEN                       sshd/3273

List all processes on Linux

python vol.py -f /mnt/ram-20181122.lime --profile Linuxamzn-4_14_72-73_55_amzn2_x86_64x64 linux_pslist
Volatility Foundation Volatility Framework 2.6
Offset             Name                 Pid             PPid            Uid             Gid    DTB                Start Time
------------------ -------------------- --------------- --------------- --------------- ------ ------------------ ----------
0xffff88001c628000 systemd              1               0               0               0      0x000000001b014000 2018-11-15 16:22:54 UTC+0000
0xffff88001c62a5c0 kthreadd             2               0               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c648000 kworker/0:0H         4               2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c64cb80 mm_percpu_wq         6               2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c690000 ksoftirqd/0          7               2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c6925c0 rcu_sched            8               2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c694b80 rcu_bh               9               2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c698000 migration/0          10              2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c69a5c0 watchdog/0           11              2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000

-- SNIP --

0xffff88000890cb80 sshd                 23161           3273            0               0      0x000000000a35c000 2018-11-22 12:56:43 UTC+0000
0xffff880019598000 sshd                 23179           23161           1000            1000   0x000000000a246000 2018-11-22 12:56:43 UTC+0000
0xffff88001679a5c0 bash                 23180           23179           1000            1000   0x0000000008990000 2018-11-22 12:56:43 UTC+0000
0xffff880017b30000 kworker/u30:0        23287           2               0               0      ------------------ 2018-11-22 12:57:16 UTC+0000
0xffff8800182525c0 sudo                 23415           23180           0               0      0x000000000a308000 2018-11-22 12:59:15 UTC+0000
0xffff880018254b80 insmod               23416           23415           0               0      0x00000000005a6000 2018-11-22 12:59:15 UTC+0000

There are other interesting possibilities , to get an idea what you can query run volatility with the -h option

python vol.py --info | grep linux_

This will list all comands, available for linux memory dumps.
Note that some commands might not work, because they are not supported for a specific profile.

Process list on Windows

volatility]$ python vol.py -f /mnt/20181122.mem  --profile Win2016x64_14393 pslist
Volatility Foundation Volatility Framework 2.6
Offset(V)          Name                    PID   PPID   Thds     Hnds   Sess  Wow64 Start                          Exit                          
------------------ -------------------- ------ ------ ------ -------- ------ ------ ------------------------------ ------------------------------
0xffffbf0b9de5e500 System                    4      0    144        0 ------      0 2018-11-18 23:32:48 UTC+0000                                 
0xffffbf0b9e6ec040 smss.exe                388      4      2        0 ------      0 2018-11-18 23:32:49 UTC+0000                                 
0xffffbf0b9e6a2080 csrss.exe               524    516      9        0      0      0 2018-11-18 23:33:00 UTC+0000                                 
0xffffbf0b9e98f080 smss.exe                584    388      0 --------      1      0 2018-11-18 23:33:01 UTC+0000                                 
0xffffbf0b9e9a7080 csrss.exe               592    584      9        0      1      0 2018-11-18 23:33:01 UTC+0000                                 
0xffffbf0b9e9f3080 wininit.exe             608    516      1        0      0      0 2018-11-18 23:33:01 UTC+0000                                 
0xffffbf0b9e9c1080 winlogon.exe            644    584      2        0      1      0 2018-11-18 23:33:01 UTC+0000                                 
0xffffbf0b9ec7b080 services.exe            704    608      4        0      0      0 2018-11-18 23:33:01 UTC+0000                                 
0xffffbf0b9ec81080 lsass.exe               712    608      7        0      0      0 2018-11-18 23:33:02 UTC+0000                                 
0xffffbf0b9ecc4380 svchost.exe             784    704     16        0      0      0 2018-11-18 23:33:03 UTC+0000                                 
0xffffbf0b9eceb840 svchost.exe             836    704     11        0      0      0 2018-11-18 23:33:03 UTC+0000

-- SNIP --

0xffffbf0b9f44e840 userinit.exe           3120   2204      0 --------      2      0 2018-11-21 22:12:45 UTC+0000                                 
0xffffbf0b9f4d5840 explorer.exe           3136   3120     60        0      2      0 2018-11-21 22:12:45 UTC+0000                                 
0xffffbf0b9e252840 TabTip.exe             3148    996     12        0      2      0 2018-11-21 22:12:45 UTC+0000                                 
0xffffbf0b9f249840 TabTip32.exe           3212   3148      1        0      2      1 2018-11-21 22:12:46 UTC+0000                                 
0xffffbf0b9f5fd840 ShellExperienc         3988    784     20        0      2      0 2018-11-21 22:12:56 UTC+0000                                 
0xffffbf0b9f44c340 SearchUI.exe           4084    784     16        0      2      0 2018-11-21 22:12:59 UTC+0000                                 
0xffffbf0b9f7795c0 MpCmdRun.exe           4568   4528      5        0      0      0 2018-11-21 22:13:08 UTC+0000                                 
0xffffbf0b9f695080 WUDFHost.exe           2376    996      6        0      0      0 2018-11-22 12:05:39 UTC+0000                                 
0xffffbf0b9f8e4840 conhost.exe            2968   3668      0 --------      2      0 2018-11-22 12:25:08 UTC+0000                                 
0xffffbf0b9f29b840 RamCapture64.e         4004   3136      9        0      2      0 2018-11-22 12:25:46 UTC+0000                                 
0xffffbf0b9e90a840 conhost.exe            3828   4004      9        0      2      0 2018-11-22 12:25:46 UTC+0000

Open ports on Windows

python vol.py -f /mnt/20181122.mem  --profile Win2016x64_14393 netscan | grep LISTEN
Volatility Foundation Volatility Framework 2.6
0xbf0b9ded6560     TCPv4    0.0.0.0:3389                   0.0.0.0:0            LISTENING        960      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xbf0b9ded6970     TCPv4    0.0.0.0:3389                   0.0.0.0:0            LISTENING        960      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xbf0b9ded6970     TCPv6    :::3389                        :::0                 LISTENING        960      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xbf0b9e8e5320     TCPv4    0.0.0.0:49666                  0.0.0.0:0            LISTENING        968      svchost.exe    2018-11-18 23:33:05 UTC+0000
0xbf0b9e9bb710     TCPv4    0.0.0.0:49666                  0.0.0.0:0            LISTENING        968      svchost.exe    2018-11-18 23:33:05 UTC+0000
0xbf0b9e9bb710     TCPv6    :::49666                       :::0                 LISTENING        968      svchost.exe    2018-11-18 23:33:05 UTC+0000
0xbf0b9eb97a30     TCPv4    0.0.0.0:49667                  0.0.0.0:0            LISTENING        1580     spoolsv.exe    2018-11-18 23:33:05 UTC+0000
0xbf0b9eb97a30     TCPv6    :::49667                       :::0                 LISTENING        1580     spoolsv.exe    2018-11-18 23:33:05 UTC+0000
0xbf0b9eb988b0     TCPv4    0.0.0.0:49667                  0.0.0.0:0            LISTENING        1580     spoolsv.exe    2018-11-18 23:33:05 UTC+0000
0xbf0b9ebd9c80     TCPv4    0.0.0.0:445                    0.0.0.0:0            LISTENING        4        System         2018-11-18 23:33:05 UTC+0000
0xbf0b9ebd9c80     TCPv6    :::445                         :::0                 LISTENING        4        System         2018-11-18 23:33:05 UTC+0000
0xbf0b9ebf9010     TCPv4    0.0.0.0:5985                   0.0.0.0:0            LISTENING        4        System         2018-11-18 23:33:05 UTC+0000
0xbf0b9ebf9010     TCPv6    :::5985                        :::0                 LISTENING        4        System         2018-11-18 23:33:05 UTC+0000
0xbf0b9ec2dc00     TCPv4    0.0.0.0:47001                  0.0.0.0:0            LISTENING        4        System         2018-11-18 23:33:05 UTC+0000
0xbf0b9ec2dc00     TCPv6    :::47001                       :::0                 LISTENING        4        System         2018-11-18 23:33:05 UTC+0000
0xbf0b9ec33ec0     TCPv4    0.0.0.0:49669                  0.0.0.0:0            LISTENING        704      services.exe   2018-11-18 23:33:05 UTC+0000
0xbf0b9ec33ec0     TCPv6    :::49669                       :::0                 LISTENING        704      services.exe   2018-11-18 23:33:05 UTC+0000
0xbf0b9ec4a8c0     TCPv4    10.100.4.174:139               0.0.0.0:0            LISTENING        4        System         2018-11-18 23:33:04 UTC+0000
0xbf0b9ecf83e0     TCPv4    0.0.0.0:135                    0.0.0.0:0            LISTENING        836      svchost.exe    2018-11-18 23:33:03 UTC+0000
0xbf0b9ecfa8f0     TCPv4    0.0.0.0:135                    0.0.0.0:0            LISTENING        836      svchost.exe    2018-11-18 23:33:03 UTC+0000
0xbf0b9ecfa8f0     TCPv6    :::135                         :::0                 LISTENING        836      svchost.exe    2018-11-18 23:33:03 UTC+0000
0xbf0b9ed06ba0     TCPv4    0.0.0.0:49664                  0.0.0.0:0            LISTENING        608      wininit.exe    2018-11-18 23:33:03 UTC+0000
0xbf0b9ed07a30     TCPv4    0.0.0.0:49664                  0.0.0.0:0            LISTENING        608      wininit.exe    2018-11-18 23:33:03 UTC+0000
0xbf0b9ed07a30     TCPv6    :::49664                       :::0                 LISTENING        608      wininit.exe    2018-11-18 23:33:03 UTC+0000
0xbf0b9edd0c00     TCPv4    0.0.0.0:49665                  0.0.0.0:0            LISTENING        528      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xbf0b9edd2a30     TCPv4    0.0.0.0:49665                  0.0.0.0:0            LISTENING        528      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xbf0b9edd2a30     TCPv6    :::49665                       :::0                 LISTENING        528      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xbf0b9ee908b0     TCPv4    0.0.0.0:49669                  0.0.0.0:0            LISTENING        704      services.exe   2018-11-18 23:33:05 UTC+0000
0xbf0b9f00ec40     TCPv4    0.0.0.0:49671                  0.0.0.0:0            LISTENING        712      lsass.exe      2018-11-18 23:33:13 UTC+0000
0xbf0b9f116c70     TCPv4    0.0.0.0:49671                  0.0.0.0:0            LISTENING        712      lsass.exe      2018-11-18 23:33:13 UTC+0000
0xbf0b9f116c70     TCPv6    :::49671                       :::0                 LISTENING        712      lsass.exe      2018-11-18 23:33:13 UTC+0000
0xd200000d6560     TCPv4    0.0.0.0:3389                   0.0.0.0:0            LISTENING        960      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xd200000d6970     TCPv4    0.0.0.0:3389                   0.0.0.0:0            LISTENING        960      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xd200000d6970     TCPv6    :::3389                        :::0                 LISTENING        960      svchost.exe    2018-11-18 23:33:04 UTC+0000

Established connections on Windows

 python vol.py -f /mnt/20181122.mem  --profile Win2016x64_14393 netscan | grep EST
Volatility Foundation Volatility Framework 2.6
0xbf0b9f385d00     TCPv4    10.100.4.174:3389              94.143.189.241:35347 ESTABLISHED      960      svchost.exe    2018-11-22 11:46:20 UTC+0000

A whole range of other commands are supported, but remember, not all commands work under all circumstances, YMMV.

Now where is that coffee?

The post Security Incident: Be Prepared – Memory Dumps appeared first on Cloudar.

The post Cloudar achieves AWS Premier Consulting Partner status appeared first on Cloudar.

Combining some bits and pieces from Google allowed me to setup Fail2Ban on the Bastion instance, while the blocking of the IPs is done in AWS NACLs in stead of the local Iptables. The setup has been done on an AmazonLinux instance.

Considerations

AWS NACLs by default only allow 20 ingress and 20 egress rules. This is a soft limit, and you can have it increased to 40 by opening a support case (40 seems to be the upper hard limit).

If fail2ban wants to add another rule while the maximum has been reached, it will block the offender in the local Iptables. Running my implementation in several environments for a few weeks now, I never had more than 4 to 5 IPs blocked at the same time. Unless you are off course the victim of a targetted ddos attack…

Setup the AWS CLI

Make sure you have a correctly installed and working AWS CLI on the instance. Also make sure the EC2 role has the necessary permissions to modify the EC2 Network ACL. Testing it out:

1. Get the current MAC address of the first interface

INTERFACE=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/)

2. Get the subnet ID from the MAC address

SUBNET_ID=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/${INTERFACE}/subnet-id)

3. Get the current Network ACL ID

ACL_ID=$(aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=$SUBNET_ID | jq '.NetworkAcls[0].Associations[0].NetworkAclId' | sed 's/"//g')

4. Test that you can add a rule to the ACL

aws ec2 create-network-acl-entry --network-acl-id $ACL_ID --ingress --rule-number 1 --protocol tcp --port-range From=0,To=65535 --cidr-block 1.2.3.4/32 --rule-action deny

5. Verify that the above IP has been blocked

aws ec2 describe-network-acls --filters Name=association.network-acl-id,Values=$ACL_ID

6. Remove the rule again

aws ec2 delete-network-acl-entry --network-acl-id $ACL_ID --ingress --rule-number 1

Fail2Ban AWS integration

1. Install the necessary packages (if not yet present)

pip install requests boto3 tabulate
yum install sqlite

2. Create a directory to store the AWS NACL script and cd to it

mkdir /opt/aws-nacl
cd /opt/aws-nacl

3. Place the following content in the file aws_nacl.py in the above directory

"""This script is used to block and unblock IPs on Amazon EC2
network ACLs and can be used with Fail2Ban. Since only 20
inbound rules are allowed with AWS if a 'jail' is provided
the IP will be blocked on the host iptables if full"""
 
import json
import sqlite3
import argparse
import os
import pprint
import socket
import logging
import logging.handlers
import requests
import boto3
import subprocess
from tabulate import tabulate
 
#Constants
#AWS only allows 20 inbound subtract default ACL rules from 20 for max
MAX_BLOCKS = 20
#Set rule start, by default AWS ACL starts rules at 100
RULE_BASE = 1
#Set range for rules: highest rule ID that can be used
RULE_RANGE = 50
 
 
def check_block(ip,acl):
   acl = get_acl(acl)
   list = acl['NetworkAcls'][0]['Entries']
   for entry in list:
      if ip in entry["CidrBlock"]:
         return True
   return False
 
 
def get_acl(acl_id):
    """This function gets the ACL given an ec2 object and ACL id"""
    ec2 = boto3.client('ec2')
    acl_response = ec2.describe_network_acls(
        NetworkAclIds=[
            acl_id,
        ],
    )
    return acl_response
 
def print_inbound_acl(acl_id):
   blocks = []
   table = {num:name[8:] for name,num in vars(socket).items() if name.startswith("IPPROTO")}
   acl = get_acl(acl_id)
   list = acl['NetworkAcls'][0]['Entries']
   for entry in list:
     if not entry["Egress"]:
         if "PortRange" in entry:
                ports = ({"To":entry["PortRange"]["To"], "From":entry["PortRange"]["From"]})
         else:
                ports = ({"To":"", "From":""})
         if entry['Protocol'] == "-1":
                proto = "all"
         else:
                proto = table[int (entry['Protocol'])]
         blocks.append([entry['RuleNumber'],proto,entry['CidrBlock'],ports["To"],ports["From"],entry['RuleAction']])
   print "Inbound Network ACL"
   print tabulate(blocks,headers=["Rule","Protocol","CIDR","Port From","Port To","Action"])
 
def is_acl(acl):
    ec2 = boto3.client('ec2')
    try:
        ec2.describe_network_acls(
            NetworkAclIds=[
                acl,
            ],
        )
        return True
    except Exception:
        return False
 
def get_acl_id():
    ec2 = boto3.client('ec2')
    meta = "http://169.254.169.254/latest/meta-data/network/interfaces/macs/"
    mac = requests.get(meta).text
    subnet = requests.get(meta+mac+"/subnet-id").text
 
    response = ec2.describe_network_acls(
        Filters=[
            {
                'Name': 'association.subnet-id',
                'Values':[
                    subnet
                ]
            },
        ],
        DryRun=False
    )
    return response['NetworkAcls'][0]['Associations'][0]['NetworkAclId']
 
def validate_ip(ip_address):
    ip_split = ip_address.split('.')
    if len(ip_split) != 4:
        return False
    for octet in ip_split:
        if not octet.isdigit():
            return False
        octet_int = int(octet)
        if octet_int < 0 or octet_int > 255:
            return False
    try:
        socket.inet_aton(ip_address)
        return True
    except socket.error:
        return False
 
def sqlite_connect(file_name):
    make_table = '''CREATE TABLE if not exists blocks (id integer PRIMARY KEY AUTOINCREMENT,
               ip text NOT NULL, acl text NOT NULL, blocked boolean NOT NULL,host boolean
               NOT NULL)'''
    if not os.path.isfile(file_name):
        dir_path = os.path.dirname(os.path.realpath(__file__))
        conn = sqlite3.connect("{}/{}".format(dir_path, file_name))
        cursor = conn.cursor()
        cursor.execute(make_table)
        conn.commit()
    else:
        try:
            conn = sqlite3.connect(file_name)
            cursor = conn.cursor()
            cursor.execute(make_table)
            conn.commit()
        except Exception:
            print "Datatbase File is encrypted or is not a database"
            exit(1)
    return conn
 
 
def main():
    logging.basicConfig(level=logging.ERROR)
    my_logger = logging.getLogger(__file__)
    my_logger.info('Checking arguments')
    parser = argparse.ArgumentParser(description="Script to block IPs on AWS EC2 Network ACL")
    parser.add_argument('-a', '--acl', help='ACL ID')
    parser.add_argument('-j', '--jail', help='Fail2Ban Jail')
    parser.add_argument('-d', '--db', default='aws-nacl.db', help='Database')
    parser.add_argument('-b', '--block', metavar="IP", help='Block IP address')
    parser.add_argument('-u', '--unblock', metavar="IP", help='Unblock IP address')
    parser.add_argument('-g', '--get', action='store_true', help='Get ACL')
    parser.add_argument('-v', '--verbose', action='store_true', help='Verbose logging')
    args = parser.parse_args()
 
    ec2_resource = boto3.resource('ec2')
    pretty_printer = pprint.PrettyPrinter(indent=4)
 
    if args.verbose:
        my_logger.info('Setting logging to debug')
        my_logger.setLevel(logging.DEBUG)
 
    if (args.block and args.unblock):
        my_logger.error('Invalid arguments')
        parser.print_usage()
        exit(1)
 
    if args.acl:
        my_logger.info('Checking if valid AWS Network ACL')
        if not is_acl(args.acl):
            print('Invalid Network ACL ID')
            my_logger.error('Invalid Network ACL')
            exit(1)
    else:
        my_logger.info('Searching for current ACL ID')
        acl = get_acl_id()
        network_acl = ec2_resource.NetworkAcl(acl)
        my_logger.debug('Network ACL ID: {}'.format(network_acl))
 
    if args.get or (not args.block and not args.unblock):
        my_logger.info('Printing ACL')
        #pretty_printer.pprint(get_acl(acl)['NetworkAcls'][0]['Entries'])
        print_inbound_acl(acl)
        exit(0)
 
 
    my_logger.info('Configuring DB')
    conn = sqlite_connect(args.db)
    cursor = conn.cursor()
 
    if args.block:
        my_logger.info('Checking if valid IP')
        if not validate_ip(args.block):
            print "IP {} is invalid".format(args.ip)
            exit(1)
        my_logger.info('Searching DB for IP: {}'.format(args.block))
        cursor.execute('''select count (*) from blocks where ip=? and blocked=1''', (args.block,))
        if cursor.fetchone()[0] > 0:
            print "IP {} already blocked".format(args.block)
            exit(0)
        my_logger.info('Checking AWS block count')
        cursor.execute('''select count (*) from blocks where blocked=1 and host =0''')
        block_count = cursor.fetchone()[0]
        my_logger.debug('Currently {} IPs blocked'.format(block_count))
        if block_count <= MAX_BLOCKS:
            my_logger.debug('Current blocks less then Max: {}'.format(MAX_BLOCKS))
            my_logger.info('Adding block to the DB')
            cursor.execute('''insert into blocks (ip, acl, blocked,host)
                               values (?,?,?,?)''', (args.block, acl, 1, 0))
            conn.commit()
            my_logger.info('Caculating Rule number based on DB ID')
            cursor.execute('''select seq from sqlite_sequence where name="blocks"''')
            rule_num = cursor.fetchone()[0] % RULE_RANGE + RULE_BASE
            my_logger.info('Adding Network ACL')
            network_acl.create_entry(
                CidrBlock=args.block+'/32',
                DryRun=False,
                Egress=False,
                PortRange={
                    'From': 0,
                    'To': 65535
                },
                Protocol='-1',
                RuleAction='deny',
                RuleNumber=rule_num
            )
            if not check_block(args.block, acl):
               my_logger.error('Failed to block IP {} in AWS ACL'.format(args.block))
               cursor.execute('''UPDATE blocks SET blocked = 0 where ip=? and
                              blocked=1''', (args.block,))
               conn.commit()
        else:
            my_logger.debug('Max blocks on AWS Network ACL, checking for IPTables')
            if  args.jail:
                my_logger.info('Blocking IP {} in f2b-{}'.format(args.block,args.jail))
                iptables = "/sbin/iptables -w -I {} 1 -s {} -j REJECT".format(args.jail, args.block)
                print iptables
                subprocess.call(iptables, shell=True)
                cursor.execute('''insert into blocks (ip, acl, blocked,host)
                                  values (?,?,?,?)''', (args.block, '', 1, 1))
                conn.commit()
            else:
                my_logger.error('No IPtables Chain set, IP will not be blocked')
    if args.unblock:
        my_logger.info('Checking if valid IP')
        if not validate_ip(args.unblock):
            my_logger.error("IP {} is invalid".format(args.unblock))
            exit(1)
        my_logger.info('Checking for IP in the DB')
        test = 'select id, host from blocks where ip="{}" and blocked=1'.format(args.unblock)
        cursor.execute(test)
        results = cursor.fetchone()
        if results is not None:
            my_logger.info('Found IP, getting rule number from DB')
            if results[1] == 0:
                rule_num = results[0] % RULE_RANGE + RULE_BASE
                my_logger.debug('Rule number is {}'.format(rule_num))
                my_logger.info('Deleting rule from AWS Network ACL')
                response = network_acl.delete_entry(
                    DryRun=False,
                    Egress=False,
                    RuleNumber=rule_num
                )
                my_logger.info('Updating DB')
                cursor.execute('''UPDATE blocks SET blocked = 0 where ip=? and
                                   blocked=1''', (args.unblock,))
                conn.commit()
            else:
                if args.jail:
                    my_logger.info('Unblocking IP {} in f2b-{}'.format(args.unblock,args.jail))
                    iptables = 'iptables -w -D {} -s {} -j REJECT'.format(args.jail, args.unblock)
                    subprocess.call(iptables, shell=True)
                    cursor.execute('''UPDATE blocks SET blocked = 0 where ip=? and blocked=1''', (args.unblock,))
                    conn.commit()
        else:
            my_logger.error("IP {} not in blocks database".format(args.unblock))
            exit(1)
 
 
if __name__ == "__main__":
    main()

4. Test the script by calling it

python aws_nacl.py -d aws-nacl.db -b 1.2.3.4 -v

5. Verify that the above IP was added to the ACL

python aws_nacl.py -g

6. Remove the IP again

python aws_nacl.py -d aws-nacl.db -u 1.2.3.4 -v

7. Verify that the IP was removed again

python aws_nacl.py -g

Fail2Ban

1. Install Fail2Ban itself

yum install fail2ban

2. Place the following file under /etc/fail2ban/action.d/aws.conf

# Fail2Ban configuration file
#
# Author: Cyril Jaquier
# Modified by Yaroslav Halchenko for multiport banning
# Modified by Ryan for AWS Network ACL block
#
 
[INCLUDES]
 
before = iptables-blocktype.conf
 
[Definition]
 
# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart = iptables -N fail2ban-<name>
              iptables -A fail2ban-<name> -j RETURN
              iptables -I <chain> -p <protocol> --dport <port> -j fail2ban-<name>
 
# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
#
actionstop = iptables -D <chain> -p <protocol> --dport <port> -j fail2ban-<name>
             iptables -F fail2ban-<name>
             iptables -X fail2ban-<name>
 
# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
#
actioncheck = iptables -n -L <chain> | grep -q 'fail2ban-<name>[ \t]'
 
# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = python /opt/aws-nacl/aws_nacl.py -d /opt/aws-nacl/aws-nacl.db -v -b <ip> -j fail2ban-<name>
# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionunban = python /opt/aws-nacl/aws_nacl.py -d /opt/aws-nacl/aws-nacl.db -v -u <ip> -j fail2ban-<name>
 
[Init]
 
# Default name of the chain
#
name = default
 
# Option:  port
# Notes.:  specifies port to monitor
# Values:  [ NUM | STRING ]  Default:
#
port = ssh
 
# Option:  protocol
# Notes.:  internally used by config reader for interpolations.
# Values:  [ tcp | udp | icmp | all ] Default: tcp
#
protocol = tcp
 
# Option:  chain
# Notes    specifies the iptables chain to which the fail2ban rules should be
#          added
# Values:  STRING  Default: INPUT
chain = INPUT

logtarget = /var/log/fail2ban.log

4. Add a file /etc/fail2ban/jail.local with the following content. Modify the values at your own convenience

[DEFAULT]
#Localhost and your office HQ range
ignoreip = 127.0.0.1/8 10.10.10.0/24

[ssh-iptables]
action = aws[name=SSH, port=ssh, protocol=tcp]

# "bantime" is the number of seconds that a host is banned.
bantime  = 3600

# "maxretry" is the number of failures before a host get banned.
maxretry = 2

5. Check /etc/fail2ban/filter.d/sshd.conf that the correct matching patterns for the /var/log/secure logfile are present, so that it actually looks for the loglines that you want to be considered as malicious. On an Amazon Linux, I have the following in place

failregex = ^%(__prefix_line)s(?:error: PAM: )?[aA]uthentication (?:failure|error) for .* from <HOST>( via \S+)?\s*$
            ^%(__prefix_line)s(?:error: PAM: )?User not known to the underlying authentication module for .* from <HOST>\s*$
            ^%(__prefix_line)sFailed \S+ for .* from <HOST>(?: port \d*)?(?: ssh\d*)?\s*$
            ^%(__prefix_line)sROOT LOGIN REFUSED.* FROM <HOST>\s*$
            ^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from <HOST> .*$
            ^%(__prefix_line)sUser .+ from <HOST> not allowed because not listed in AllowUsers\s*$
            ^%(__prefix_line)sUser .+ from <HOST> not allowed because listed in DenyUsers\s*$
            ^%(__prefix_line)sUser .+ from <HOST> not allowed because not in any group\s*$
            ^%(__prefix_line)srefused connect from \S+ \(<HOST>\)\s*$
            ^%(__prefix_line)sUser .+ from <HOST> not allowed because a group is listed in DenyGroups\s*$
            ^%(__prefix_line)sUser .+ from <HOST> not allowed because none of user's groups are listed in AllowGroups\s*$
            ^%(__prefix_line)sReceived disconnect from <HOST> port \d*:11: Bye Bye \[preauth\]

6. Add fail2ban to the startup list and start it

chkconfig --add fail2ban
chkconfig fail2ban on
service fail2ban start

Locked myself out

Things can always go wrong, but thanks to the recently released AWS feature called ‘SSM Session Manager’ you can always get a console window on your instance to start troubleshooting. One thing to make sure is that your instance is running the latest version of the AWS SSM Agent. So it’s always a good idea to update it before closing your current SSH session:

yum install https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm

You also need to make sure that your instance is allowed to communicate with the AWS SSM service. Easiest way is to attach the ‘AmazonEC2RoleforSSM‘ policy to your EC2 role.

Credits

https://github.com/TheRemover/fail2ban-aws-nacl

https://techbytesecurity.com/2017/06/fail2ban-with-aws-network-acl

https://www.google.com

The post Integrating Fail2Ban with AWS Network ACLs appeared first on Cloudar.

Amazon EC2 Systems Manager is a collection of capabilities that helps you automate management tasks such as collecting system inventory, applying operating system patches, automating the creation of Amazon Machine Images (AMIs), and configuring operating systems and applications at scale. It is available at no cost to manage both your EC2 and on-premises resources!

Amazon EC2 Systems Manager relies on the Amazon Simple Systems Management Service (SSM) agent being installed on the guests. The SSM agent is pre-installed on Windows Server 2016 instances or Windows Server 2003-2012 R2 instances created from AMI’s published after November 2016. You need at least SSM agent version 2.0.599.0 installed on the target EC2 instance.

In this article we will focus on using Systems Manager to apply Windows Updates to EC2 instances. Patch Management is always an operational pain point so its welcome that AWS offers a solution.

You start by creating groups of instances by applying a tag called ‘Patch Group’. Then you create a group of patches by forming a patch baseline containing and excluding the patches you require (or use the AWS default patch baseline). At last you create a maintenance window to have your patch baseline attached and applied to a patch group. The actual ‘Patch Now’ run-command is nothing more than an API call, so there’s no obligation to use Maintenance Windows. Personally I’m a fan of Rundeck, so I’ll show you how to have the patches applied to the instances using both methods.

Configure your instances

The guest SSM agent setting inside with Windows OS requires permissions to connect to AWS EC2 Systems Manager. We grant these rights by creating an EC2 Service Role with the policy document ‘AmazonEC2RoleforSSM’ attached. Then you can attach this role to your instances. The instance also needs outbound internet connection to be able to connect to SSM. This can be either through an Internet Gateway or a NAT Gateway (or NAT Instance).

If you have this done right, your instance(s) should pop-up under ‘Managed Instances’ in the EC2 console:

Take note of the SSM Agent Version. As said earlier it must be at least version 2.0.599.0. The Systems Manager Service also requires a “Patch Group”-tag on the EC2 instance. The key for a patch group tag must be Patch Group. Note that the key is case sensitive. The value can be anything you want to specify, but the key must be Patch Group.

If done correctly, your tag will be picked up by SSM. You can confirm this on the ‘Managed Instances’ page:

Patch Baselines

AWS provides a default Patch Baseline called ‘AWS-DefaultPatchBaseline’. It auto-approves all critical and security updates with a ‘critical’ or ‘important’ classification seven days after they have been released by Microsoft. If you’re happy with that you can use this baseline. If you’re not, you can simply create your own according to your requirements: set approval for specific products and patch classifications, exclude a specific KB etc

Once your happy with your baseline, you can hit ‘Create’. Now assign it to one or more Patch Groups (or make it the default baseline and throw away the AWS one). Hit the ‘actions’ menu and chose ‘Modify Patch Groups’

Type the names of the Patch Groups you defined when tagging your instances

Your baseline is now attached to the specified patch groups. You can now start evaluating your instances against the baseline, and update them accordingly.

Patching

Applying the patch baseline to a specific instance or to a patch group is nothing more than executing an AWS SSM run command. You can schedule this run command through AWS SSM ‘Maintenance Windows’, a cron job on a server (like Rundeck) or manual through the AWS Console.

Let’s first check everything manually. In the AWS EC2 console, go to ‘Run Commands’ and create a new Run Command. Select the ‘AWS-ApplyPatchBaseline’ command document and pick an instance run this on. For the ‘operation’, choose ‘Scan’. This will evaluate the instance against the baseline without installing anything yet.

Once the run command finishes, you can go back to the ‘Managed Instances’ page. Highlight the instance(s) on which the run command was executed and click on the ‘Patch’ tab. Here you can see the result of the scan:

To actually install the missing updates, execute the same run command document, but now with the ‘Install’ operation. This will install the missing KBs to the instances and reboot them if needed.

Or execute the following aws cli command to accomplish the same:

aws ssm send-command --targets "Key=tag:Patch Group,Values=<PatchGroupName>" --document-name "AWS-ApplyPatchBaseline" --comment "Install|Check Windows Updates" --parameters Operation="<Install|Scan>"

Maintenance Windows

In stead of manually starting a run command or cron job, we can also use the AWS provided Maintenance Windows feature. Systems Manager Maintenance Windows let you define a schedule for when to perform actions on your instances such as patching the operating system. Each Maintenance Window has a schedule, a duration, a set of registered targets, and a set of registered tasks.

Before actually creating a Maintenance Window, we must configure a Maintenance Window role. We need this so Systems Manager can execute tasks in Maintenance Windows on our behalf. So we go to the IAM page and create a new role. We pick an “EC2 service role” type and make sure to attach the “AmazonSSMMaintenanceWindowRole” policy to it. Once the role is created, we must modify it. Click “edit Trust Relationships”. Add a comma after “ec2.amazonaws.com”, and then add “Service”: “ssm.amazonaws.com” to the existing policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com",
        "Service": "ssm.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Back to SSM now to actually create the Maintenance Window. Give it a useful name and specify your preferred schedule. I’m setting ‘every 30 minutes’ just for demonstration purposes, but in a real setup you would most probably choose something like ‘Every  Sunday’. You can also configure your own Cron expression.

This leaves us now with an empty Maintenance Window: there are no tasks nor targets associated yet.

To assign targets to the Maintenance Window, click on the “Register new targets” button on the “Targets” tab. We dynamically select the targets by using the “Patch Group” tag.

We will now have an ID linked to our “dev” Patch Group. This “Window Target ID” is used in the next step.

From the “tasks” tab of the Maintenance Window, click on “Schedule new task”. Pick the “AWS-ApplyPatchBaseline” document. Under “Registered Targets”, select the correct Window Target ID. For the operation, select “Install”. For the “Role”, select the IAM role with the AmazonSSMMaintenanceWindowRole attached to it (the one we created earlier). Set your preferred concurrency level and register the task by clicking on the blue button. The end result should look like this:

Now we have to wait for the schedule of the Maintenance Window. In this example we specified ‘every 30 minutes’ as a schedule, so the waiting shouldn’t take too long. Under the ‘History’ tab of the Maintenance Window you can follow all actions. The Maintenance Window will simply launch a Run Command, so you could go to that console screen too. If you enabled logging to S3, you could find the output of the Run Command over there. If not, you can view a (truncated) output via the Run Command itself:

Patch Summary for i-07ca5621af38f256d
PatchGroup          : dev
BaselineId          : pb-06101e06cf8506be6
SnapshotId          : 317f2b72-2612-4740-95af-c7b3d8fb6d1e
OwnerInformation    : 
OperationType       : Install
OperationStartTime  : 2017-05-29T11:00:14.0000000Z
OperationEndTime    : 2017-05-29T11:03:18.7164313Z
InstalledCount      : 1
InstalledOtherCount : 6
FailedCount         : 0
MissingCount        : 0
NotApplicableCount  : 3

EC2AMAZ-EA5SH8I - PatchBaselineOperations Installation Results - 2017-05-29T11:03:19.537

KbArticleId Installed   Message
----------- ----------- -----------
KB890830    Yes         Success

If we now go back to the “Managed Instances” page and look at the “Patch” tab of our test instance, we will see it is not missing any updates anymore!

Success! Another  on the Automation checklist!

Rutger

The post Windows servers patching with AWS EC2 Systems Manager appeared first on Cloudar.

• First, enable the “Monthly report” to be able to receive a detailed report of your AWS usage.
• Secondly, you’ll need to enable “Receive Billing Reports” and create an Amazon S3 Bucket in which estimated and monthly billing reports can be stored
• Once the Amazon S3 Bucket has been created, you must apply appropriate permissions to your S3 bucket through a Bucket Policy to allow reports to be published. A policy grants AWS access to publish your report to your bucket.
  To add the policy to your bucket, you will need to sign in to the AWS Console and add permissions to your bucket. For more information, please review Editing Bucket PermissionsBelow is an Amazon S3 Bucket sample policy:

{
  "Version": "2008-10-17",
  "Id": "Policy1335892530063",
  "Statement": [
    {
      "Sid": "Stmt1335892150622",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::386209384616:root"
      },
      "Action": [
        "s3:GetBucketAcl",
        "s3:GetBucketPolicy"
      ],
      "Resource": "arn:aws:s3:::cloudar-production"
    },
    {
      "Sid": "Stmt1335892526596",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::386209384616:root"
      },
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::cloudar-production/*"
    }
  ]
}

Once you created your S3 Bucket, you’ll be able to activate “receive billing reports” and verify your S3 bucket policy. When your S3 bucket is validated, you can save your preferences and AWS will deliver these detailed billing reports to your Amazon S3 Bucket.

The post AWS detailed billing with resources and tags appeared first on Cloudar.