ACL Archives | Cloudar

Security Incident: Be Prepared – Memory Dumps

Koenraad de Boevé — Sun, 25 Nov 2018 08:43:45 +0000

Memory Dumps

You just finished setting up your super-duper AWS environment..

Highly available & Fault Tolerant: check!
Backups in place: check!
MFA enforced: check!
Security Groups and NACLs: check!
CloudTrail enabled: check!

You even deserve bonus points for activating Amazon GuardDuty and putting AWS WAF & Shield in front of your CloudFront distribution and loadbalancers.
Time to lean back with a bit of smugness, while you take a sip of your well-deserved cup of coffee.

Seems like you have covered all your bases, or have you?

One often overlooked topic is security incident response.
While a lot of security incidents can (and should) be mitigated through automation,
some incidents will require manual intervention, such as information and evidence gathering during and after a successful malicious attack on one of your instances.

Effective incident response and forensics require preparation, well ahead of time.
It is critical to have your forensics and remediation tools readily available for whenever the proverbial shit hits the fan.
Documenting your investigative steps and being able to execute them swiftly, will contribute to a well-controlled, thorough and effective investigation.

In this blog post, I’d like to focus on some of the first steps you might take in your investigation:

Building a forensics workstation and taking a memory dump of a compromised instance.
Preparation steps and tools, both for windows and linux.
The forensics investigation process.
An investigation of a real memory dump.

Although some great tools are already available from Threadresponse (such as AWS_IR and Margarita Shotgun), we will build a solution ourselves, in order to gain a thorough knowledge on how the process works behind the scenes.
Creating a memory dump is something that should be done immediately, as it provides a snapshot of the memory at the time of the attack.
The dump can then be analyzed and used to build a timeline, improve security after the fact and optionally provide evidence for any follow up with law enforcement.

Tools

Linux: LiME ( Linux Memory Extractor )
Windows: FTK or Belkasoft Live RamCapturer
( both are free, but require registration, you get the download link in a mail )
Volatility

Preparation Steps

Create Quarantine and Forensic Security Groups

Forensics Security Group.
This SG will be attached to your Forensics Workstation later on.

aws ec2 create-security-group --group-name ForensicsSG \
--description "Forensics SG" --vpc-id  \
--profile

This will output something like this:

{
    "GroupId": "sg-22222222222222222"
}

Now we use this GroupId to add an inbound rule that allows SSH connection towards our Forensics Workstation

aws ec2 authorize-security-group-ingress --group-id sg-22222222222222222 \
--protocol tcp --port 22 --cidr  --profile 
By default , outbound everything is allowed, and in this case, that is ok, so we leave it at that.

Quarantine Security Group
This SG will be attached to the compromised instance, to isolate it from any network, except the forensics network.

aws ec2 create-security-group --group-name QuarantineSG \
--description "Quarantine SG" --vpc-id  \
--profile

Output will look similar to the following:

{
    "GroupId": "sg-111111111111111111"
}

Remove all rules from outbound ( egress )

aws ec2 revoke-security-group-egress --group-id sg-11111111111111111 \
--ip-permissions '[{"IpProtocol": "-1","IpRanges": [{"CidrIp": "0.0.0.0/0"}],"Ipv6Ranges": [{"CidrIpv6": "::/0"}]}]' \
--profile

Add rules to allow access from the Forensics Security Group

aws ec2 authorize-security-group-ingress --group-id sg-11111111111111111 \
--ip-permissions '[
    {"IpProtocol":"tcp","FromPort":22,"ToPort":22,"UserIdGroupPairs":[
        {"GroupId": "sg-22222222222222222","Description": "SSH access from the ForensicsSG"}
    ]},
    {"IpProtocol":"tcp","FromPort":4444,"ToPort":4444,"UserIdGroupPairs":[
        {"GroupId":"sg-22222222222222222","Description":"Access from the ForensicsSG for LiME dump over TCP"}
    ]},
    {"IpProtocol":"tcp","FromPort":3389,"ToPort":3389,"UserIdGroupPairs":[
        {"GroupId":"sg-22222222222222222","Description":"RDP access from the ForensicsSG"}
    ]}
    ]' \
--profile

Create Isolation Functionality
Create lambda-execution-forensics-trust-policy.json file with following content:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create the Lambda Execution role, forensics-lambda-exec-role, using the policy file you just created:

aws iam create-role \
--role-name forensics-lambda-exec-role  \
--assume-role-policy-document file://lambda-execution-forensics-trust-policy.json \
--profile

The output should look like this:

{
    "Role": {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17", 
            "Statement": [
                {
                    "Action": "sts:AssumeRole", 
                    "Effect": "Allow", 
                    "Principal": {
                        "Service": "lambda.amazonaws.com"
                    }
                }
            ]
        }, 
        "RoleId": "AROAILWIMFHDIIGTDST7I", 
        "CreateDate": "2018-11-20T20:42:44Z", 
        "RoleName": "forensics-lambda-exec-role", 
        "Path": "/", 
        "Arn": "arn:aws:iam::111111111111:role/forensics-lambda-exec-role"
    }
}

Create lambda-forensics-policy.json with following content:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:CreateLogGroup"
            ],
            "Resource": "arn:aws:logs:*:*:*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "ec2:Describe*",
                "ec2:ModifyNetworkInterfaceAttribute",
                "ec2:ModifyInstanceAttribute"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Create the policy:

aws iam create-policy --policy-name lambda-execute-forensics-policy \
--policy-document file://lambda-forensics-policy.json 
--profile

This will output something like this:

{
    "Policy": {
        "PolicyName": "lambda-execute-forensics-policy", 
        "PermissionsBoundaryUsageCount": 0, 
        "CreateDate": "2018-11-20T20:56:04Z", 
        "AttachmentCount": 0, 
        "IsAttachable": true, 
        "PolicyId": "ANPAIG23L72JCRWAQQEFE", 
        "DefaultVersionId": "v1", 
        "Path": "/", 
        "Arn": "arn:aws:iam::111111111111:policy/lambda-execute-forensics-policy", 
        "UpdateDate": "2018-11-20T20:56:04Z"
    }
}

Attach the policy to the forensics-lambda-exec-role, using the ARN found in the output from the previous comand

aws iam attach-role-policy --role-name forensics-lambda-exec-role \
--policy-arn arn:aws:iam::111111111111:policy/lambda-execute-forensics-policy \
--profile

Create index.py with following content:

import json
import boto3
import os

ec2 = boto3.resource('ec2')
quarantine_sg = os.environ['QUARANTINE_SG']

def set_SecurityGroup(instance):
  oldsecuritygroups = {}
  interfaces = instance.network_interfaces
  for interface in interfaces:
    oldsecuritygroups[interface.id] =  interface.groups
    interface.modify_attribute(Groups = [quarantine_sg])

  return oldsecuritygroups

def lambda_handler(event, context):
  instance = ec2.Instance(event['instance_id'])
  # Remove current Security Groups
  orig_groups = set_SecurityGroup(instance)
  return {
    'statusCode': 200,
    'body': "OK",
    'replaced_sgs': orig_groups
  }

This function will replace all Security Groups on all attached network interfaces of an instance with the Quarantine Security Group we created earlier. When you discover that an instance is compromised, you can invoke this function, and it will isolate the instance. It fetches the Security Group from the environment, and takes the instance id of the compromised instance as input as json in the following format:

{
    "instance_id": "i-xxxxxxxxxxxxxxxxx"
}

The function will output the removed Security Groups so you have a reference for later on.

zip the file to index.zip

zip index.zip index.py

Create the Lambda function, using the IAM Policy ARN from the output of the role creation (forensics-lambda-exec-role)

aws lambda create-function --function-name forensics-isolate-instance \
--zip-file fileb://index.zip \
--role arn:aws:iam::111111111111:role/forensics-lambda-exec-role \
--handler index.lambda_handler --runtime python3.6 \
--environment Variables={QUARANTINE_SG=sg-11111111111111111} \
--profile

OK, Done, now let s move on and create a forensics workstation.

Build a Forensics Workstation

Create EC2 Instance Role for the forensics Workstation
Create ec2-forensics-trust-policy.json file with following content:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create the EC2 Instance role, ec2-forensics-role, using the Policy file you just created:

aws iam create-role --role-name ec2-forensics-role \
--assume-role-policy-document file://ec2-forensics-trust-policy.json \
--profile

Output:

{
    "Role": {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17", 
            "Statement": [
                {
                    "Action": "sts:AssumeRole", 
                    "Effect": "Allow", 
                    "Principal": {
                        "Service": "ec2.amazonaws.com"
                    }
                }
            ]
        }, 
        "RoleId": "AROAIGXSH7RCMPCVPY4QA", 
        "CreateDate": "2018-11-21T10:34:07Z", 
        "RoleName": "ec2-forensics-role", 
        "Path": "/", 
        "Arn": "arn:aws:iam::111111111111:role/ec2-forensics-role"
    }
}

Create ec2-forensics-policy.json with following content:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEC2InstanceEC2Forensics",
            "Effect": "Allow",
            "Action": [
                "ec2:Describe*",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:ModifyVolumeAttribute",
                "ec2:CreateKeyPair",
                "ec2:ReportInstanceStatus",
                "ec2:ModifySnapshotAttribute",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:ImportKeyPair",
                "ec2:CreateTags",
                "ec2:StopInstances",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:AttachVolume",
                "ec2:ImportVolume",
                "ec2:ModifySubnetAttribute",
                "ec2:CreateSnapshot",
                "ec2:RebootInstances",
                "ec2:ImportInstance",
                "ec2:ResetSnapshotAttribute",
                "ec2:ImportSnapshot",
                "ec2:CopySnapshot",
                "ec2:CreateImage",
                "ec2:CopyImage",
                "ec2:GetLaunchTemplateData",
                "ec2:ImportImage",
                "ec2:DetachVolume",
                "ec2:CreateFlowLogs",
                "ec2:GetConsoleOutput",
                "ec2:CreateSecurityGroup",
                "ec2:CreateNetworkAcl",
                "ec2:ModifyInstanceAttribute",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:DetachNetworkInterface",
                "ec2:CreateNetworkAclEntry"
            ],
            "Resource": "*"
        },
        {
            "Sid": "AllowEC2InstanceInvokeLambda",
            "Effect": "Allow",
            "Action": "lambda:InvokeFunction",
            "Resource": "arn:aws:lambda:eu-west-1:111111111111:function:forensics-isolate-instance"
        }
    ]
}

Create the ec2-forensics-policy IAM Policy:

aws iam create-policy --policy-name ec2-forensics-policy \
--policy-document file://ec2-forensics-policy.json \
--profile

Attach the policy to the ec2-forensics-role, using the ARN found in the output from the previous command.
The Forensics Workstation also needs access to S3 and AWS Systems Manager, so we also include some AWS Managed Policies

aws iam attach-role-policy --role-name ec2-forensics-role \
--policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM \
--profile 
aws iam attach-role-policy --role-name ec2-forensics-role \
--policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess \
--profile 
aws iam attach-role-policy --role-name ec2-forensics-role \
--policy-arn arn:aws:iam::111111111111:policy/ec2-forensics-policy \
--profile

Create an Instance Profile, called ec2-forensics-profile

aws iam create-instance-profile \
--instance-profile-name ec2-forensics-profile \
--profile

Attach the ec2-forensics-role to the ec2-forensics-profile

aws iam add-role-to-instance-profile \
--instance-profile-name ec2-forensics-profile \
--role-name ec2-forensics-role \
--profile

Provision the Forensics Workstation
Create user-data.txt script with following content:

#!/bin/bash

# Install prerequisites
sudo yum -y update
sudo yum install python-pip pcre-tools gcc autoconf automake libtool nc git kernel-devel libdwarf-tools
pip install distorm3 pycrypto pillow openpyxl ujson pytz IPython

# Install Volatility
cd /home/ec2-user
wget http://downloads.volatilityfoundation.org/releases/2.6/volatility-2.6.zip
unzip volatility-2.6.zip
mv volatility-master volatility
chown -R ec2-user.ec2-user volatility

# Install LiME
git clone https://github.com/504ensicsLabs/LiME.git
chown -R ec2-user.ec2-user LiME

Create the EC2 Instance

aws ec2 run-instances --image-id ami-09693313102a30b2c \
--count 1 --instance-type t3.micro --key-name MyKeyPair \
--security-group-ids sg-22222222222222222 \
--subnet-id subnet-33333333333333333 \
--user-data file://user-data.txt \
--tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value="My Forensics WS"}]' \
--iam-instance-profile Name=ec2-forensics-profile \
--profile

Tip: You can now fetch the latest Amazon AMI by querying the SSM Parameter Store:

aws ssm get-parameters \
--names /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 \
--profile 
{
    "InvalidParameters": [], 
    "Parameters": [
        {
            "Name": "/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2", 
            "LastModifiedDate": 1542668357.322, 
            "Value": "ami-09693313102a30b2c", 
            "Version": 11, 
            "Type": "String", 
            "ARN": "arn:aws:ssm:eu-west-1::parameter/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"
        }
    ]
}

We now have a basic workstation.

Create a forensic volume
We will need a volume we can attach/detach from any instance to temporarily store memory dumps and provide tools to the compromised instance.
Since we need to support both Windows and Linux, we create an exfat filesystem on it, as it is supported on all platforms.

aws ec2 create-volume --availability-zone eu-west-1a \ 
--size 100 --volume-type gp2 --tag-specifications \
'ResourceType=volume,Tags=[{Key=Name,Value=exFat-Forensics-Volume}]' \
--profile

This creates a 100 GB volume. You can lower the size , but make sure that it is comfortably bigger then the largest ram size on any of your instances.

Connect to the Forensics Workstation and attach the new volume ( you probably need to run aws configure to set your default region first )

aws ec2 attach-volume --device /dev/sdX \
--volume-id  \
--instance-id $(curl http://169.254.169.254/latest/meta-data/instance-id)

Partition and format the disk

sudo fdisk /dev/sdh

Welcome to fdisk (util-linux 2.30.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x601015b4.

Command (m for help): p
Disk /dev/sdh: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x601015b4

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-209715199, default 2048): 
Last sector, +sectors or +size{K,M,G,T,P} (2048-209715199, default 209715199): 

Created a new partition 1 of type 'Linux' and of size 100 GiB.

Command (m for help): t
Selected partition 1          
Hex code (type L to list all codes): 7
Changed type of partition 'Linux' to 'HPFS/NTFS/exFAT'.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

sudo mkfs.vfat /dev/sdh1

Mounting the volume

sudo mount /dev/sdhX /mnt

Copy your local LiME and Volatility folders to the volume, to be able to compile a kernel module or create a Volatility profile on the fly, just in case.
For Windows, we copy RamCapturer or/and FTK Imager to the volume as well.

Once you have all tools on the volume, unmount it, detach it and make a snapshot.

Next step, we will gather LiME kernel modules and create Volatility Profiles for instances, running in our environment.
The LiME kernel module needs to be loaded on the compromised instance

Create LiME kernel modules and Volatility Profiles

Note that you need to compile the kernel module for the EXACT kernel version , running on your instances. If you don’t patch your systems and/or have a lot of different Linux flavors, you will have a hard time maintaining the lime kernel modules and volatility profiles.

You have 2 options in obtaining a lime kernel module matching your kernel:

threadresponse lime module repository
https://threatresponse-lime-modules.s3.amazonaws.com/
This opens an xmlfile in which the available lime modules can be found
Here is an excerpt of that xml file.
```
threatresponse-lime-modules


<-- SNIP -->


modules/lime-2.6.32-131.0.15.el6.centos.plus.x86_64.ko

2018-09-06T18:40:37.000Z
"93efeac2519a4c6a573601d203416098"
1098692
STANDARD



modules/lime-2.6.32-131.0.15.el6.centos.plus.x86_64.ko.sig

2018-09-06T18:41:34.000Z
"c4d44af3b2265e55e8c23b8dc62d8828"
566
STANDARD

..
```
The reference to the kernel modules ( in the above example on line 8 ) is the s3 object key to the object.
So if your kernel is 2.6.32-131.0.15.el6, you can download it from https://threatresponse-lime-modules.s3.amazonaws.com/modules/lime-2.6.32-131.0.15.el6.centos.plus.x86_64.ko
build your own repository
If you cannot find the lime modules for kernels running in your environment, you can build your own.
Install volatility and LiME on either an existing instance, or launch a new instance of the Linux distribution for which you want to create a module and volatility profile.For Redhat/CentOS/ Amazon Linux, you can use the userdata from the Forensics Workstation (2 .Provision the Forensics Workstation)
For Debian/Ubuntu you can use this user-data content:
```
#!/bin/bash
sudo apt -y update
sudo apt -y install libelf-dev libdwarf-dev dwarfdump zip make gcc
cd /home/ubuntu
wget http://downloads.volatilityfoundation.org/releases/2.6/volatility-2.6.zip
unzip volatility.zip
mv volatility-master volatility
chown -R ubuntu.ubuntu volatility
# Install LiME
git clone https://github.com/504ensicsLabs/LiME.git
chown -R ubuntu.ubuntu LiME
```
1. Compile the LiME module
  First, Make sure you have kernel headers, source and images installed on this instance.
```
cd ~/LiME/src
make
```
  This should have created a file called lime-$(uname -r).ko
  If you need to create a module for a different kernel version ( for example for an older unpatched instance ),
  install the required version and change the above command to:
```
make KVER=
```
  Note that this does not require the targeted kernel version to be active
2. Create Volatility Profiles
```
cd ~/volatility/tools/linux

# change os to reflect your distro
os="ubuntu"
# change kernel version to the kernel version you want to compile for
# do make sure this kernel is installed ( including headers, source and boot image
# however, the kernel version does not need to be actively running
kernel_version=$(uname -r)
make KVER=${kernel_version}
zip ~/${os}-${kernel_version}.zip module.dwarf /boot/System.map-${kernel_version}
```
3. Copy the lime module (.ko file) and volatility profile (.zip file) to an S3 Bucket that acts as a central repository.

Incident Workflow

Logon onto the Forensics WorkStation
ssh into your Forensics Workstation in 2 terminal windows. ( Use ssh-agent, ssh-add and ssh -A)

Isolate the compromised instance

aws lambda invoke --function-name forensics-isolate-instance \
--payload '{"instance_id": ""}' /tmp/output.txt

Create a snapshot of the compromised instance

aws ec2 create-snapshot \
--volume-id  \
--tag-specifications \
'ResourceType=snapshot,Tags=[{Key=Name,Value=compromised-instance-snap}]'

Attach the Forensics Volume to the compromised EC2 Instance and mount it.
On The Forensics Workstation:
```
aws ec2 attach-volume --device /dev/sdX \
--volume-id  \
--instance-id 
```
On the compromised EC2 Instance:

Linux:
ssh into the compromised instance, fetch the kernel version and mount the volume.
We need the kernel version for the next steps
```
uname -r
sudo mount /dev/sdX1 /mnt
```
Windows:
RDP into the compromised instance ( you need to setup an SSH Tunnel via the Forensics Workstation )
Mount the volume via Computer Management -> Disk Management

The new Volume is attached and identified as Disk 1 , but it is in an offline state.
Right-click on Disk 1 and select ‘Online’

The Volume is now online and mapped to D:

On Windows, you can now skip to Step 7
In the other terminal ( Forensics WS ), fetch the lime kernel module from your S3 Bucket or the Threadresponse repository
scp the LiME module to the compromised instance
Run the memory dump on the compromised instance
On Linux:
```
sudo insmod /path/to/lime-$(uname -r).ko "path=/mnt/ram.lime format=lime digest=sha1"
```
This will create the memory dump file ram.lime and the digest file ram.sha1 on the forensics volume.

On Windows:
Open File Explorer, and go to D:\
If RamCapturer is not yet unzipped, unzip RamCapturer.zip first.
Then run D:\RamCapturer\x64\RamCapturer.exe as Administrator

Save the dump to D:\ and run ‘Capture!’
The dump will be saved as YYYYMMDD.mem where YYYYMMDD is the current date.
Fetch the memory dump onto the Forensics WorkStation
unmount the Forensics Volume on Linux, or , on Windows, put it offline again using the Disk Management.
Detach the volume from the compromised EC2 Instance and attach it back to the Forensics Workstation.
Stop the Compromised instance.

Sample Results from Memory Dumps

Volatility requires a profile matching your kernel. For Windows this is already included , but for Linux , you might need the volatility profile to be imported into your volatility setup.
let’s first test if the profile for your kernel is already configured:

cd volatility
python vol.py --info | grep Profile
Volatility Foundation Volatility Framework 2.6
Profiles
Linuxamzn-4_14_72-73_55_amzn2_x86_64x64 - A Profile for Linux amzn-4.14.72-73.55.amzn2.x86_64 x64
Linuxamzn-4_14_77-81_59_amzn2_x86_64x64 - A Profile for Linux amzn-4.14.77-81.59.amzn2.x86_64 x64
VistaSP0x64                             - A Profile for Windows Vista SP0 x64
VistaSP0x86                             - A Profile for Windows Vista SP0 x86
VistaSP1x64                             - A Profile for Windows Vista SP1 x64
VistaSP1x86                             - A Profile for Windows Vista SP1 x86
VistaSP2x64                             - A Profile for Windows Vista SP2 x64
VistaSP2x86                             - A Profile for Windows Vista SP2 x86
Win10x64                                - A Profile for Windows 10 x64
Win10x64_10240_17770                    - A Profile for Windows 10 x64 (10.0.10240.17770 / 2018-02-10)

...

If your kernel is not listed, you can add it by copying the volatility profile ( created in section 2.2 Create Volatility Profile )
to the volatility/plugins/overlays/linux/ directory.
Rerunning the above command should show your added profile.

OK, Pièce de résistance: some results:

Fetching lsof for the linux memory dump:

python vol.py -f /mnt/ram-20181122.lime \
--profile Linuxamzn-4_14_72-73_55_amzn2_x86_64x64 linux_lsof

Volatility Foundation Volatility Framework 2.6
Offset             Name                           Pid      FD       Path
------------------ ------------------------------ -------- -------- ----
0xffff88001c628000 systemd                               1        0 /dev/null
0xffff88001c628000 systemd                               1        1 /dev/null
0xffff88001c628000 systemd                               1        2 /dev/null
0xffff88001c628000 systemd                               1        3 anon_inode:[6744]
0xffff88001c628000 systemd                               1        4 anon_inode:[6744]
0xffff88001c628000 systemd                               1        5 anon_inode:[6744]
0xffff88001c628000 systemd                               1        6 /sys/fs/cgroup/systemd
0xffff88001c628000 systemd                               1        7 anon_inode:[6744]
0xffff88001c628000 systemd                               1        8 socket:[14109]
0xffff88001c628000 systemd                               1        9 /proc/1/mountinfo

-- SNIP --

0xffff8800182525c0 sudo                              23415        3 pipe:[1758814]
0xffff8800182525c0 sudo                              23415        4 pipe:[1758814]
0xffff8800182525c0 sudo                              23415        5 socket:[1758834]
0xffff8800182525c0 sudo                              23415        6 socket:[1758839]
0xffff880018254b80 insmod                            23416        0 /dev/pts/0
0xffff880018254b80 insmod                            23416        1 /dev/pts/0
0xffff880018254b80 insmod                            23416        2 /dev/pts/0
0xffff880018254b80 insmod                            23416        3 /mnt/lime-modules/amazon/lime-4.14.72-73.55.amzn2.x86_64.ko

List all established connections on Linux

python vol.py -f /mnt/ram-20181122.lime --profile Linuxamzn-4_14_72-73_55_amzn2_x86_64x64 linux_netstat | grep EST
Volatility Foundation Volatility Framework 2.6
TCP      10.100.4.6      :   22 10.100.4.111    :44854 ESTABLISHED                  sshd/23161
TCP      10.100.4.6      :   22 10.100.4.111    :44854 ESTABLISHED                  sshd/23179

List all open ports on Linux

python vol.py -f /mnt/ram-20181122.lime --profile Linuxamzn-4_14_72-73_55_amzn2_x86_64x64 linux_netstat | grep LISTEN
Volatility Foundation Volatility Framework 2.6
TCP      0.0.0.0         :  111 0.0.0.0         :    0 LISTEN                    rpcbind/2662 
TCP      ::              :  111 ::              :    0 LISTEN                    rpcbind/2662 
TCP      127.0.0.1       :   25 0.0.0.0         :    0 LISTEN                     master/3143 
TCP      0.0.0.0         :   22 0.0.0.0         :    0 LISTEN                       sshd/3273 
TCP      ::              :   22 ::              :    0 LISTEN                       sshd/3273

List all processes on Linux

python vol.py -f /mnt/ram-20181122.lime --profile Linuxamzn-4_14_72-73_55_amzn2_x86_64x64 linux_pslist
Volatility Foundation Volatility Framework 2.6
Offset             Name                 Pid             PPid            Uid             Gid    DTB                Start Time
------------------ -------------------- --------------- --------------- --------------- ------ ------------------ ----------
0xffff88001c628000 systemd              1               0               0               0      0x000000001b014000 2018-11-15 16:22:54 UTC+0000
0xffff88001c62a5c0 kthreadd             2               0               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c648000 kworker/0:0H         4               2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c64cb80 mm_percpu_wq         6               2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c690000 ksoftirqd/0          7               2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c6925c0 rcu_sched            8               2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c694b80 rcu_bh               9               2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c698000 migration/0          10              2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000
0xffff88001c69a5c0 watchdog/0           11              2               0               0      ------------------ 2018-11-15 16:22:54 UTC+0000

-- SNIP --

0xffff88000890cb80 sshd                 23161           3273            0               0      0x000000000a35c000 2018-11-22 12:56:43 UTC+0000
0xffff880019598000 sshd                 23179           23161           1000            1000   0x000000000a246000 2018-11-22 12:56:43 UTC+0000
0xffff88001679a5c0 bash                 23180           23179           1000            1000   0x0000000008990000 2018-11-22 12:56:43 UTC+0000
0xffff880017b30000 kworker/u30:0        23287           2               0               0      ------------------ 2018-11-22 12:57:16 UTC+0000
0xffff8800182525c0 sudo                 23415           23180           0               0      0x000000000a308000 2018-11-22 12:59:15 UTC+0000
0xffff880018254b80 insmod               23416           23415           0               0      0x00000000005a6000 2018-11-22 12:59:15 UTC+0000

There are other interesting possibilities , to get an idea what you can query run volatility with the -h option

python vol.py --info | grep linux_

This will list all comands, available for linux memory dumps.
Note that some commands might not work, because they are not supported for a specific profile.

Process list on Windows

volatility]$ python vol.py -f /mnt/20181122.mem  --profile Win2016x64_14393 pslist
Volatility Foundation Volatility Framework 2.6
Offset(V)          Name                    PID   PPID   Thds     Hnds   Sess  Wow64 Start                          Exit                          
------------------ -------------------- ------ ------ ------ -------- ------ ------ ------------------------------ ------------------------------
0xffffbf0b9de5e500 System                    4      0    144        0 ------      0 2018-11-18 23:32:48 UTC+0000                                 
0xffffbf0b9e6ec040 smss.exe                388      4      2        0 ------      0 2018-11-18 23:32:49 UTC+0000                                 
0xffffbf0b9e6a2080 csrss.exe               524    516      9        0      0      0 2018-11-18 23:33:00 UTC+0000                                 
0xffffbf0b9e98f080 smss.exe                584    388      0 --------      1      0 2018-11-18 23:33:01 UTC+0000                                 
0xffffbf0b9e9a7080 csrss.exe               592    584      9        0      1      0 2018-11-18 23:33:01 UTC+0000                                 
0xffffbf0b9e9f3080 wininit.exe             608    516      1        0      0      0 2018-11-18 23:33:01 UTC+0000                                 
0xffffbf0b9e9c1080 winlogon.exe            644    584      2        0      1      0 2018-11-18 23:33:01 UTC+0000                                 
0xffffbf0b9ec7b080 services.exe            704    608      4        0      0      0 2018-11-18 23:33:01 UTC+0000                                 
0xffffbf0b9ec81080 lsass.exe               712    608      7        0      0      0 2018-11-18 23:33:02 UTC+0000                                 
0xffffbf0b9ecc4380 svchost.exe             784    704     16        0      0      0 2018-11-18 23:33:03 UTC+0000                                 
0xffffbf0b9eceb840 svchost.exe             836    704     11        0      0      0 2018-11-18 23:33:03 UTC+0000

-- SNIP --

0xffffbf0b9f44e840 userinit.exe           3120   2204      0 --------      2      0 2018-11-21 22:12:45 UTC+0000                                 
0xffffbf0b9f4d5840 explorer.exe           3136   3120     60        0      2      0 2018-11-21 22:12:45 UTC+0000                                 
0xffffbf0b9e252840 TabTip.exe             3148    996     12        0      2      0 2018-11-21 22:12:45 UTC+0000                                 
0xffffbf0b9f249840 TabTip32.exe           3212   3148      1        0      2      1 2018-11-21 22:12:46 UTC+0000                                 
0xffffbf0b9f5fd840 ShellExperienc         3988    784     20        0      2      0 2018-11-21 22:12:56 UTC+0000                                 
0xffffbf0b9f44c340 SearchUI.exe           4084    784     16        0      2      0 2018-11-21 22:12:59 UTC+0000                                 
0xffffbf0b9f7795c0 MpCmdRun.exe           4568   4528      5        0      0      0 2018-11-21 22:13:08 UTC+0000                                 
0xffffbf0b9f695080 WUDFHost.exe           2376    996      6        0      0      0 2018-11-22 12:05:39 UTC+0000                                 
0xffffbf0b9f8e4840 conhost.exe            2968   3668      0 --------      2      0 2018-11-22 12:25:08 UTC+0000                                 
0xffffbf0b9f29b840 RamCapture64.e         4004   3136      9        0      2      0 2018-11-22 12:25:46 UTC+0000                                 
0xffffbf0b9e90a840 conhost.exe            3828   4004      9        0      2      0 2018-11-22 12:25:46 UTC+0000

Open ports on Windows

python vol.py -f /mnt/20181122.mem  --profile Win2016x64_14393 netscan | grep LISTEN
Volatility Foundation Volatility Framework 2.6
0xbf0b9ded6560     TCPv4    0.0.0.0:3389                   0.0.0.0:0            LISTENING        960      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xbf0b9ded6970     TCPv4    0.0.0.0:3389                   0.0.0.0:0            LISTENING        960      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xbf0b9ded6970     TCPv6    :::3389                        :::0                 LISTENING        960      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xbf0b9e8e5320     TCPv4    0.0.0.0:49666                  0.0.0.0:0            LISTENING        968      svchost.exe    2018-11-18 23:33:05 UTC+0000
0xbf0b9e9bb710     TCPv4    0.0.0.0:49666                  0.0.0.0:0            LISTENING        968      svchost.exe    2018-11-18 23:33:05 UTC+0000
0xbf0b9e9bb710     TCPv6    :::49666                       :::0                 LISTENING        968      svchost.exe    2018-11-18 23:33:05 UTC+0000
0xbf0b9eb97a30     TCPv4    0.0.0.0:49667                  0.0.0.0:0            LISTENING        1580     spoolsv.exe    2018-11-18 23:33:05 UTC+0000
0xbf0b9eb97a30     TCPv6    :::49667                       :::0                 LISTENING        1580     spoolsv.exe    2018-11-18 23:33:05 UTC+0000
0xbf0b9eb988b0     TCPv4    0.0.0.0:49667                  0.0.0.0:0            LISTENING        1580     spoolsv.exe    2018-11-18 23:33:05 UTC+0000
0xbf0b9ebd9c80     TCPv4    0.0.0.0:445                    0.0.0.0:0            LISTENING        4        System         2018-11-18 23:33:05 UTC+0000
0xbf0b9ebd9c80     TCPv6    :::445                         :::0                 LISTENING        4        System         2018-11-18 23:33:05 UTC+0000
0xbf0b9ebf9010     TCPv4    0.0.0.0:5985                   0.0.0.0:0            LISTENING        4        System         2018-11-18 23:33:05 UTC+0000
0xbf0b9ebf9010     TCPv6    :::5985                        :::0                 LISTENING        4        System         2018-11-18 23:33:05 UTC+0000
0xbf0b9ec2dc00     TCPv4    0.0.0.0:47001                  0.0.0.0:0            LISTENING        4        System         2018-11-18 23:33:05 UTC+0000
0xbf0b9ec2dc00     TCPv6    :::47001                       :::0                 LISTENING        4        System         2018-11-18 23:33:05 UTC+0000
0xbf0b9ec33ec0     TCPv4    0.0.0.0:49669                  0.0.0.0:0            LISTENING        704      services.exe   2018-11-18 23:33:05 UTC+0000
0xbf0b9ec33ec0     TCPv6    :::49669                       :::0                 LISTENING        704      services.exe   2018-11-18 23:33:05 UTC+0000
0xbf0b9ec4a8c0     TCPv4    10.100.4.174:139               0.0.0.0:0            LISTENING        4        System         2018-11-18 23:33:04 UTC+0000
0xbf0b9ecf83e0     TCPv4    0.0.0.0:135                    0.0.0.0:0            LISTENING        836      svchost.exe    2018-11-18 23:33:03 UTC+0000
0xbf0b9ecfa8f0     TCPv4    0.0.0.0:135                    0.0.0.0:0            LISTENING        836      svchost.exe    2018-11-18 23:33:03 UTC+0000
0xbf0b9ecfa8f0     TCPv6    :::135                         :::0                 LISTENING        836      svchost.exe    2018-11-18 23:33:03 UTC+0000
0xbf0b9ed06ba0     TCPv4    0.0.0.0:49664                  0.0.0.0:0            LISTENING        608      wininit.exe    2018-11-18 23:33:03 UTC+0000
0xbf0b9ed07a30     TCPv4    0.0.0.0:49664                  0.0.0.0:0            LISTENING        608      wininit.exe    2018-11-18 23:33:03 UTC+0000
0xbf0b9ed07a30     TCPv6    :::49664                       :::0                 LISTENING        608      wininit.exe    2018-11-18 23:33:03 UTC+0000
0xbf0b9edd0c00     TCPv4    0.0.0.0:49665                  0.0.0.0:0            LISTENING        528      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xbf0b9edd2a30     TCPv4    0.0.0.0:49665                  0.0.0.0:0            LISTENING        528      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xbf0b9edd2a30     TCPv6    :::49665                       :::0                 LISTENING        528      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xbf0b9ee908b0     TCPv4    0.0.0.0:49669                  0.0.0.0:0            LISTENING        704      services.exe   2018-11-18 23:33:05 UTC+0000
0xbf0b9f00ec40     TCPv4    0.0.0.0:49671                  0.0.0.0:0            LISTENING        712      lsass.exe      2018-11-18 23:33:13 UTC+0000
0xbf0b9f116c70     TCPv4    0.0.0.0:49671                  0.0.0.0:0            LISTENING        712      lsass.exe      2018-11-18 23:33:13 UTC+0000
0xbf0b9f116c70     TCPv6    :::49671                       :::0                 LISTENING        712      lsass.exe      2018-11-18 23:33:13 UTC+0000
0xd200000d6560     TCPv4    0.0.0.0:3389                   0.0.0.0:0            LISTENING        960      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xd200000d6970     TCPv4    0.0.0.0:3389                   0.0.0.0:0            LISTENING        960      svchost.exe    2018-11-18 23:33:04 UTC+0000
0xd200000d6970     TCPv6    :::3389                        :::0                 LISTENING        960      svchost.exe    2018-11-18 23:33:04 UTC+0000

Established connections on Windows

 python vol.py -f /mnt/20181122.mem  --profile Win2016x64_14393 netscan | grep EST
Volatility Foundation Volatility Framework 2.6
0xbf0b9f385d00     TCPv4    10.100.4.174:3389              94.143.189.241:35347 ESTABLISHED      960      svchost.exe    2018-11-22 11:46:20 UTC+0000

A whole range of other commands are supported, but remember, not all commands work under all circumstances, YMMV.

Now where is that coffee?

The post Security Incident: Be Prepared – Memory Dumps appeared first on Cloudar.

Integrating Fail2Ban with AWS Network ACLs

Rutger Beyen — Fri, 05 Oct 2018 16:17:29 +0000

I was recently working on a project where I couldn’t lock down the Bastion instance security group ingress rule to only allow whitelisted IP addresses. Several coworkers work from home and use the Bastion to jump into backend servers and create SSH tunnels, while they did not have AWS Console access to whitelist themselves. The security group ended up allowing 0.0.0.0/0 on port 22. Off course the Bastion instance would only allow ssh key based logins, but all protocols eventually have vulnerabilities. So installing multiple layers of security in depth prevents the systems from being directly affected by the latest individual vulnerabilities.

Combining some bits and pieces from Google allowed me to setup Fail2Ban on the Bastion instance, while the blocking of the IPs is done in AWS NACLs in stead of the local Iptables. The setup has been done on an AmazonLinux instance.

Considerations

AWS NACLs by default only allow 20 ingress and 20 egress rules. This is a soft limit, and you can have it increased to 40 by opening a support case (40 seems to be the upper hard limit).

If fail2ban wants to add another rule while the maximum has been reached, it will block the offender in the local Iptables. Running my implementation in several environments for a few weeks now, I never had more than 4 to 5 IPs blocked at the same time. Unless you are off course the victim of a targetted ddos attack…

Setup the AWS CLI

Make sure you have a correctly installed and working AWS CLI on the instance. Also make sure the EC2 role has the necessary permissions to modify the EC2 Network ACL. Testing it out:

1. Get the current MAC address of the first interface

INTERFACE=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/)

2. Get the subnet ID from the MAC address

SUBNET_ID=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/${INTERFACE}/subnet-id)

3. Get the current Network ACL ID

ACL_ID=$(aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=$SUBNET_ID | jq '.NetworkAcls[0].Associations[0].NetworkAclId' | sed 's/"//g')

4. Test that you can add a rule to the ACL

aws ec2 create-network-acl-entry --network-acl-id $ACL_ID --ingress --rule-number 1 --protocol tcp --port-range From=0,To=65535 --cidr-block 1.2.3.4/32 --rule-action deny

5. Verify that the above IP has been blocked

aws ec2 describe-network-acls --filters Name=association.network-acl-id,Values=$ACL_ID

6. Remove the rule again

aws ec2 delete-network-acl-entry --network-acl-id $ACL_ID --ingress --rule-number 1

Fail2Ban AWS integration

1. Install the necessary packages (if not yet present)

pip install requests boto3 tabulate
yum install sqlite

2. Create a directory to store the AWS NACL script and cd to it

mkdir /opt/aws-nacl
cd /opt/aws-nacl

3. Place the following content in the file aws_nacl.py in the above directory

"""This script is used to block and unblock IPs on Amazon EC2
network ACLs and can be used with Fail2Ban. Since only 20
inbound rules are allowed with AWS if a 'jail' is provided
the IP will be blocked on the host iptables if full"""
 
import json
import sqlite3
import argparse
import os
import pprint
import socket
import logging
import logging.handlers
import requests
import boto3
import subprocess
from tabulate import tabulate
 
#Constants
#AWS only allows 20 inbound subtract default ACL rules from 20 for max
MAX_BLOCKS = 20
#Set rule start, by default AWS ACL starts rules at 100
RULE_BASE = 1
#Set range for rules: highest rule ID that can be used
RULE_RANGE = 50
 
 
def check_block(ip,acl):
   acl = get_acl(acl)
   list = acl['NetworkAcls'][0]['Entries']
   for entry in list:
      if ip in entry["CidrBlock"]:
         return True
   return False
 
 
def get_acl(acl_id):
    """This function gets the ACL given an ec2 object and ACL id"""
    ec2 = boto3.client('ec2')
    acl_response = ec2.describe_network_acls(
        NetworkAclIds=[
            acl_id,
        ],
    )
    return acl_response
 
def print_inbound_acl(acl_id):
   blocks = []
   table = {num:name[8:] for name,num in vars(socket).items() if name.startswith("IPPROTO")}
   acl = get_acl(acl_id)
   list = acl['NetworkAcls'][0]['Entries']
   for entry in list:
     if not entry["Egress"]:
         if "PortRange" in entry:
                ports = ({"To":entry["PortRange"]["To"], "From":entry["PortRange"]["From"]})
         else:
                ports = ({"To":"", "From":""})
         if entry['Protocol'] == "-1":
                proto = "all"
         else:
                proto = table[int (entry['Protocol'])]
         blocks.append([entry['RuleNumber'],proto,entry['CidrBlock'],ports["To"],ports["From"],entry['RuleAction']])
   print "Inbound Network ACL"
   print tabulate(blocks,headers=["Rule","Protocol","CIDR","Port From","Port To","Action"])
 
def is_acl(acl):
    ec2 = boto3.client('ec2')
    try:
        ec2.describe_network_acls(
            NetworkAclIds=[
                acl,
            ],
        )
        return True
    except Exception:
        return False
 
def get_acl_id():
    ec2 = boto3.client('ec2')
    meta = "http://169.254.169.254/latest/meta-data/network/interfaces/macs/"
    mac = requests.get(meta).text
    subnet = requests.get(meta+mac+"/subnet-id").text
 
    response = ec2.describe_network_acls(
        Filters=[
            {
                'Name': 'association.subnet-id',
                'Values':[
                    subnet
                ]
            },
        ],
        DryRun=False
    )
    return response['NetworkAcls'][0]['Associations'][0]['NetworkAclId']
 
def validate_ip(ip_address):
    ip_split = ip_address.split('.')
    if len(ip_split) != 4:
        return False
    for octet in ip_split:
        if not octet.isdigit():
            return False
        octet_int = int(octet)
        if octet_int < 0 or octet_int > 255:
            return False
    try:
        socket.inet_aton(ip_address)
        return True
    except socket.error:
        return False
 
def sqlite_connect(file_name):
    make_table = '''CREATE TABLE if not exists blocks (id integer PRIMARY KEY AUTOINCREMENT,
               ip text NOT NULL, acl text NOT NULL, blocked boolean NOT NULL,host boolean
               NOT NULL)'''
    if not os.path.isfile(file_name):
        dir_path = os.path.dirname(os.path.realpath(__file__))
        conn = sqlite3.connect("{}/{}".format(dir_path, file_name))
        cursor = conn.cursor()
        cursor.execute(make_table)
        conn.commit()
    else:
        try:
            conn = sqlite3.connect(file_name)
            cursor = conn.cursor()
            cursor.execute(make_table)
            conn.commit()
        except Exception:
            print "Datatbase File is encrypted or is not a database"
            exit(1)
    return conn
 
 
def main():
    logging.basicConfig(level=logging.ERROR)
    my_logger = logging.getLogger(__file__)
    my_logger.info('Checking arguments')
    parser = argparse.ArgumentParser(description="Script to block IPs on AWS EC2 Network ACL")
    parser.add_argument('-a', '--acl', help='ACL ID')
    parser.add_argument('-j', '--jail', help='Fail2Ban Jail')
    parser.add_argument('-d', '--db', default='aws-nacl.db', help='Database')
    parser.add_argument('-b', '--block', metavar="IP", help='Block IP address')
    parser.add_argument('-u', '--unblock', metavar="IP", help='Unblock IP address')
    parser.add_argument('-g', '--get', action='store_true', help='Get ACL')
    parser.add_argument('-v', '--verbose', action='store_true', help='Verbose logging')
    args = parser.parse_args()
 
    ec2_resource = boto3.resource('ec2')
    pretty_printer = pprint.PrettyPrinter(indent=4)
 
    if args.verbose:
        my_logger.info('Setting logging to debug')
        my_logger.setLevel(logging.DEBUG)
 
    if (args.block and args.unblock):
        my_logger.error('Invalid arguments')
        parser.print_usage()
        exit(1)
 
    if args.acl:
        my_logger.info('Checking if valid AWS Network ACL')
        if not is_acl(args.acl):
            print('Invalid Network ACL ID')
            my_logger.error('Invalid Network ACL')
            exit(1)
    else:
        my_logger.info('Searching for current ACL ID')
        acl = get_acl_id()
        network_acl = ec2_resource.NetworkAcl(acl)
        my_logger.debug('Network ACL ID: {}'.format(network_acl))
 
    if args.get or (not args.block and not args.unblock):
        my_logger.info('Printing ACL')
        #pretty_printer.pprint(get_acl(acl)['NetworkAcls'][0]['Entries'])
        print_inbound_acl(acl)
        exit(0)
 
 
    my_logger.info('Configuring DB')
    conn = sqlite_connect(args.db)
    cursor = conn.cursor()
 
    if args.block:
        my_logger.info('Checking if valid IP')
        if not validate_ip(args.block):
            print "IP {} is invalid".format(args.ip)
            exit(1)
        my_logger.info('Searching DB for IP: {}'.format(args.block))
        cursor.execute('''select count (*) from blocks where ip=? and blocked=1''', (args.block,))
        if cursor.fetchone()[0] > 0:
            print "IP {} already blocked".format(args.block)
            exit(0)
        my_logger.info('Checking AWS block count')
        cursor.execute('''select count (*) from blocks where blocked=1 and host =0''')
        block_count = cursor.fetchone()[0]
        my_logger.debug('Currently {} IPs blocked'.format(block_count))
        if block_count <= MAX_BLOCKS:
            my_logger.debug('Current blocks less then Max: {}'.format(MAX_BLOCKS))
            my_logger.info('Adding block to the DB')
            cursor.execute('''insert into blocks (ip, acl, blocked,host)
                               values (?,?,?,?)''', (args.block, acl, 1, 0))
            conn.commit()
            my_logger.info('Caculating Rule number based on DB ID')
            cursor.execute('''select seq from sqlite_sequence where name="blocks"''')
            rule_num = cursor.fetchone()[0] % RULE_RANGE + RULE_BASE
            my_logger.info('Adding Network ACL')
            network_acl.create_entry(
                CidrBlock=args.block+'/32',
                DryRun=False,
                Egress=False,
                PortRange={
                    'From': 0,
                    'To': 65535
                },
                Protocol='-1',
                RuleAction='deny',
                RuleNumber=rule_num
            )
            if not check_block(args.block, acl):
               my_logger.error('Failed to block IP {} in AWS ACL'.format(args.block))
               cursor.execute('''UPDATE blocks SET blocked = 0 where ip=? and
                              blocked=1''', (args.block,))
               conn.commit()
        else:
            my_logger.debug('Max blocks on AWS Network ACL, checking for IPTables')
            if  args.jail:
                my_logger.info('Blocking IP {} in f2b-{}'.format(args.block,args.jail))
                iptables = "/sbin/iptables -w -I {} 1 -s {} -j REJECT".format(args.jail, args.block)
                print iptables
                subprocess.call(iptables, shell=True)
                cursor.execute('''insert into blocks (ip, acl, blocked,host)
                                  values (?,?,?,?)''', (args.block, '', 1, 1))
                conn.commit()
            else:
                my_logger.error('No IPtables Chain set, IP will not be blocked')
    if args.unblock:
        my_logger.info('Checking if valid IP')
        if not validate_ip(args.unblock):
            my_logger.error("IP {} is invalid".format(args.unblock))
            exit(1)
        my_logger.info('Checking for IP in the DB')
        test = 'select id, host from blocks where ip="{}" and blocked=1'.format(args.unblock)
        cursor.execute(test)
        results = cursor.fetchone()
        if results is not None:
            my_logger.info('Found IP, getting rule number from DB')
            if results[1] == 0:
                rule_num = results[0] % RULE_RANGE + RULE_BASE
                my_logger.debug('Rule number is {}'.format(rule_num))
                my_logger.info('Deleting rule from AWS Network ACL')
                response = network_acl.delete_entry(
                    DryRun=False,
                    Egress=False,
                    RuleNumber=rule_num
                )
                my_logger.info('Updating DB')
                cursor.execute('''UPDATE blocks SET blocked = 0 where ip=? and
                                   blocked=1''', (args.unblock,))
                conn.commit()
            else:
                if args.jail:
                    my_logger.info('Unblocking IP {} in f2b-{}'.format(args.unblock,args.jail))
                    iptables = 'iptables -w -D {} -s {} -j REJECT'.format(args.jail, args.unblock)
                    subprocess.call(iptables, shell=True)
                    cursor.execute('''UPDATE blocks SET blocked = 0 where ip=? and blocked=1''', (args.unblock,))
                    conn.commit()
        else:
            my_logger.error("IP {} not in blocks database".format(args.unblock))
            exit(1)
 
 
if __name__ == "__main__":
    main()

4. Test the script by calling it

python aws_nacl.py -d aws-nacl.db -b 1.2.3.4 -v

5. Verify that the above IP was added to the ACL

python aws_nacl.py -g

6. Remove the IP again

python aws_nacl.py -d aws-nacl.db -u 1.2.3.4 -v

7. Verify that the IP was removed again

python aws_nacl.py -g

Fail2Ban

1. Install Fail2Ban itself

yum install fail2ban

2. Place the following file under /etc/fail2ban/action.d/aws.conf

# Fail2Ban configuration file
#
# Author: Cyril Jaquier
# Modified by Yaroslav Halchenko for multiport banning
# Modified by Ryan for AWS Network ACL block
#
 
[INCLUDES]
 
before = iptables-blocktype.conf
 
[Definition]
 
# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart = iptables -N fail2ban-
              iptables -A fail2ban- -j RETURN
              iptables -I  -p  --dport  -j fail2ban-
 
# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
#
actionstop = iptables -D  -p  --dport  -j fail2ban-
             iptables -F fail2ban-
             iptables -X fail2ban-
 
# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
#
actioncheck = iptables -n -L  | grep -q 'fail2ban-[ \t]'
 
# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = python /opt/aws-nacl/aws_nacl.py -d /opt/aws-nacl/aws-nacl.db -v -b  -j fail2ban-
# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionunban = python /opt/aws-nacl/aws_nacl.py -d /opt/aws-nacl/aws-nacl.db -v -u  -j fail2ban-
 
[Init]
 
# Default name of the chain
#
name = default
 
# Option:  port
# Notes.:  specifies port to monitor
# Values:  [ NUM | STRING ]  Default:
#
port = ssh
 
# Option:  protocol
# Notes.:  internally used by config reader for interpolations.
# Values:  [ tcp | udp | icmp | all ] Default: tcp
#
protocol = tcp
 
# Option:  chain
# Notes    specifies the iptables chain to which the fail2ban rules should be
#          added
# Values:  STRING  Default: INPUT
chain = INPUT

3. Modify /etc/fail2ban/fail2ban.conf and change the logfile location (it’s easier to have a separate log rather then searching through /var/log/messages)

logtarget = /var/log/fail2ban.log

4. Add a file /etc/fail2ban/jail.local with the following content. Modify the values at your own convenience

[DEFAULT]
#Localhost and your office HQ range
ignoreip = 127.0.0.1/8 10.10.10.0/24

[ssh-iptables]
action = aws[name=SSH, port=ssh, protocol=tcp]

# "bantime" is the number of seconds that a host is banned.
bantime  = 3600

# "maxretry" is the number of failures before a host get banned.
maxretry = 2

5. Check /etc/fail2ban/filter.d/sshd.conf that the correct matching patterns for the /var/log/secure logfile are present, so that it actually looks for the loglines that you want to be considered as malicious. On an Amazon Linux, I have the following in place

failregex = ^%(__prefix_line)s(?:error: PAM: )?[aA]uthentication (?:failure|error) for .* from ( via \S+)?\s*$
            ^%(__prefix_line)s(?:error: PAM: )?User not known to the underlying authentication module for .* from \s*$
            ^%(__prefix_line)sFailed \S+ for .* from (?: port \d*)?(?: ssh\d*)?\s*$
            ^%(__prefix_line)sROOT LOGIN REFUSED.* FROM \s*$
            ^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from  .*$
            ^%(__prefix_line)sUser .+ from  not allowed because not listed in AllowUsers\s*$
            ^%(__prefix_line)sUser .+ from  not allowed because listed in DenyUsers\s*$
            ^%(__prefix_line)sUser .+ from  not allowed because not in any group\s*$
            ^%(__prefix_line)srefused connect from \S+ \(\)\s*$
            ^%(__prefix_line)sUser .+ from  not allowed because a group is listed in DenyGroups\s*$
            ^%(__prefix_line)sUser .+ from  not allowed because none of user's groups are listed in AllowGroups\s*$
            ^%(__prefix_line)sReceived disconnect from  port \d*:11: Bye Bye \[preauth\]

6. Add fail2ban to the startup list and start it

chkconfig --add fail2ban
chkconfig fail2ban on
service fail2ban start

Locked myself out

Things can always go wrong, but thanks to the recently released AWS feature called ‘SSM Session Manager’ you can always get a console window on your instance to start troubleshooting. One thing to make sure is that your instance is running the latest version of the AWS SSM Agent. So it’s always a good idea to update it before closing your current SSH session:

yum install https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm

You also need to make sure that your instance is allowed to communicate with the AWS SSM service. Easiest way is to attach the ‘AmazonEC2RoleforSSM‘ policy to your EC2 role.

Credits

https://github.com/TheRemover/fail2ban-aws-nacl

https://techbytesecurity.com/2017/06/fail2ban-with-aws-network-acl

https://www.google.com

The post Integrating Fail2Ban with AWS Network ACLs appeared first on Cloudar.