1. How the Internet evolved and became more hostile
As the Internet evolved from a small interconnect between universities where every resource had a public IP address, and resources were managed based on mutual respect and consensus to the Internet as we know it today with multiple threads, risks, and shortage of IP addresses, the direct and public accessibility of all resources could no longer be sustained.
As stated in the security pillar of the Well Architected Framework, the main security principle in AWS Cloud is to create several network layers and defense in depth.
The framework states more specifically: “Do not place resources in public subnets of your VPC unless they absolutely must receive inbound network traffic from public sources” SEC05-BP01 Create network layers.
So as a best practice, cloud engineers strive to deploy their resources in private subnets whenever possible. Because of the confidential nature of these subnets, engineers had to invent ways to manage their resources in these private subnets.
2. How to Use Jump / Bastion Hosts to Manage Your Private Subnets
A solution to manage the infrastructure in your private subnets is to deploy a bastion host. Before 2018, this was the recommended way, and it was well-documented:
For Linux Workloads:
https://aws.amazon.com/solutions/implementations/linux-bastion/
For Windows Workloads:
https://aws.amazon.com/solutions/implementations/rd-gateway/
Although these solutions do solve the problem stated above (how to manage your Linux and/or Windows workloads in their private subnets), these solutions have several drawbacks:
- These solutions still need a public IP address to be associated with the virtual machines and open ports to enable remote access.
- The Linux solution is vulnerable to SSH multiplexing access.
- As with all software that is publicly accessible, they need to be strictly maintained and patched:
- It’s hard to capture the remote access’s metadata and/or full session logs.
- Password management and access control are needed for the management users.
Because IT is all about answering questions, we are looking for an answer to this question: ‘How can I manage my resources in private subnets in a secure, encrypted, and auditable fashion?’ Luckily, AWS came up with two solutions that answer this question.
3. AWS Systems Manager’s New Session Manager
AWS listened to the drawbacks inherent to using bastion/jump hosts and introduced the Session Manager service as a component of AWS Systems Manager. The Session Manager supports both inside AWS and on-premises:
- Windows
- Linux
- MacOS
This blog post will focus on using it with Elastic Compute Cloud (EC2) Linux instances running in AWS.
3.1 Why AWS Session Manager
Session Manager solves the problem mentioned above: to access instances within private subnets that don’t allow ingress connections from the Internet. This is made possible by the Systems Manager (SSM) agent running on the EC2 instances, which pushes traffic to the Session Manager service managed by AWS outside your EC2 environment. The configuration shown in the diagram below provides an example of how it can be configured:
As you can see in the diagram above, the client machine is outside of AWS. It accesses the internal machines over the Internet using valid user credentials for the AWS environment. To access the internal machines, the client can use several methods:
Via the AWS console, you have two-fold access:
- The AWS Systems Manager console includes access to all the Session Manager capabilities for both administrators and end users.
- The Amazon EC2 console allows end users to connect to the EC2 instances for which they have been granted session permissions.
• Via the AWS Command Line Interface (CLI): this requires the installation of the Session Manager Plugin. It is possible to restrict how users launch sessions, so you could choose only to allow them to invoke a connection via the AWS CLI if your users don’t have access to the console or the other way around.
• The Session Manager SDK consists of libraries and sample code that allows application developers to build front-end applications, such as custom shells or self-service portals for internal users that natively use Session Manager to connect to managed nodes. This allows you to integrate Session Manager into your own tooling or automation workflows.
3.2 Configuration
3.2.1 Network Considerations
As shown in the diagram above, the SSM agent actually uses HTTPS to communicate with the SSM endpoint. As such, port 22 for SSH or port 3389 for RDP are no longer required and should not be allowed as ingress traffic. To make this even more secure, we can use VPC endpoints: they use AWS PrivateLink, so traffic between the target EC2 instances and the Session Manager service does not traverse the Internet. To enable this, create these VPC endpoints in your target VPCs, replacing the region accordingly, for example, eu-west-1
Interface endpoints:
-
com.amazonaws.[region].ssm
-
com.amazonaws.[region].ssmmessages
-
com.amazonaws.[region].ec2messages
-
com.amazonaws.[region].ec2 [For Windows workloads]
Optional but recommended:
-
com.amazonaws.[region].kms = To encrypt logs in CloudWatch
-
com.amazonaws.[region].logs = To send logs to CloudWatch
Gateway endpoint:
-
com.amazonaws.region.s3
Systems Manager uses this endpoint to update SSM Agent and to perform patching operations. Systems Manager also uses this endpoint for tasks like uploading output logs that you choose to store in S3 buckets, retrieving scripts or other files you store in buckets, and so on.
If you restrict egress networking on the EC2 machines via security groups, you should use the prefix list of the S3 gateway endpoint to send traffic to this gateway endpoint only. Here is an example of security group configuration for a tightly controlled EC2 instance:
As the interface endpoints replace the DNS addresses of the SSM services, the network traffic with the configuration listed above only flows over AWS PrivateLink. You can even further limit access to the S3 buckets that SSM uses by using the structure listed in the AWS documentation: S3 bucket list
3.2.2 EC2 Prerequisites and IAM Configuration
EC2 Prerequisites
The EC2 instance should run the Systems Manager (SSM) agent to support the Systems Manager. The Amazon Linux AMIs support the agent by default, and it can be installed manually on other Linux systems by installing the Systems Manager. The agent allows a lot more than just connecting with Session Manager. More information about its capabilities is available here. Even if you want to use it on a system that is not currently supported, you can always download the open-source code and modify it yourself. However, AWS will not support a modified version of the agent.
IAM Configuration
Each EC2 Instance must have permission to make API calls to the Session Manager Service. All necessary permissions are available in the Amazon-managed IAM policy: AmazonSSMManagedInstanceCore, and this policy should be attached to an Instance profile.
The policy listed above gives you a good baseline, but if you want to lock this down further, you can look up the information here.
Using this method to access the instances does not require any management of SSH keys. Access is only based on IAM policies since everything is managed through the Session Manager.
When configuring Session Manager access for your end-users, you can limit which instances they can access, whether it’s through the command line or the console, and if they are allowed to tunnel SSH or just use the regular Session Manager shell access.
More information about how to configure the IAM policies is available here.
Some AWS policy templates show how to restrict access to specific EC2 instances, but that will probably not be scalable if you really want to implement this for your organization.
Instead, you can refer to the templates available here to see how to configure the policies to allow access based on tags. As shown here, you will also want to ensure you add permissions for users to terminate only the sessions they started. Otherwise, a user may be able to terminate another user’s session.
3.3. Logging
3.3.1. What the AWS CloudTrail Does
Without configuring anything, AWS CloudTrail will collect basic information about sessions. When someone launches a remote access session with Session Manager, SSM will log an event named “StartSession.” This event will include a number of interesting things, such as:
- The username that launched the session
- Whether the user was authenticated with multi-factor authentication (MFA)
- The originating IP address
- The unique ID of the target EC2 instance
- The session ID
The events in CloudTrail are helpful for getting a sense of who’s logging into your instances and when it’s happening. However, it does not provide any information about what they are doing once they’ve established a session.
3.3.2. Session logs
As mentioned in the overview of AWS Session Manager, SSM uses HTTPS to establish sessions, but it is also possible to use it as a tunnel for other protocols, such as SSH or RDP. Be aware that AWS cannot provide full session logs for those connections if you use the tunneling option. The diagram below shows an example configuration for logging full session data either to CloudWatch or S3 when tunneling is not used:
The logs in CloudWatch look like this:
Next to the CloudWatch logs, logging into S3 is also possible.
Please make sure that you attach a policy to your instance profile that allows writing to the S3 bucket you configured to store your session data. The result is that you will see log files written to the specified bucket with the name of the session (the username and a unique string). The logs contain the commands typed, along with their output, which is the same as what is shown in CloudWatch.
3.4. Conclusion
The question: “How can I manage my resources in private subnets in a secure, encrypted, and auditable fashion?” is answered by using the Systems Manager Sessions Manager.
4. Instance Connect and Instance Connect Endpoint
As there are often multiple ways to reach the same goal, AWS delivers a different option to answer the same question: “How can I manage my resources in private subnets in a secure, encrypted, and auditable fashion.” This can also be solved by using Instance Connect with its newest addition: Instance Connect Endpoint
4.1. What Is Instance Connect?
The Instance Connect feature was introduced in 2019 and offers a solution to control SSH access to your instances using AWS Identity and Access Management (IAM) policies and audit connection requests with AWS CloudTrail events. In addition, you can leverage your existing SSH keys or further enhance your security posture by generating one-time use SSH keys each time an authorized user connects.
Instance Connect works with any SSH client, but you can also easily connect to your instances from the EC2 Console.
The question: “How can I manage my resources in private subnets in a secure, encrypted, and auditable fashion?” is not solved by this approach because:
- The machine still needs a public IP address.
- The connection logs cannot be stored in S3 or CloudWatch.
A few months ago, AWS introduced a new Instance Connect Endpoint feature that solves the first drawback listed above.
4.2. Instance Connect Endpoint
With the Endpoint for Instance Connect (EIC) Service, you no longer need a public IP address on your resource or any agent to connect to your resources. The Instance Connect Endpoint Service works in three ways:
- AWS Management Console
- AWS Command Line Interface (AWS CLI)
- SSH Clients like PuTTY or OpenSSH
4.2.1. Instance Connect Endpoint Features
The EIC Endpoint Service is an identity-aware TCP proxy. As described above, it has two modes:
- AWS CLI | SSH CLients
- AWS Console
The AWS CLI client is used to create a secure WebSocket tunnel from your workstation to the endpoint, where authorization happens with your AWS Identity and Access Management (IAM) credentials. When the tunnel is established, point your preferred SSH client at your loopback address (127.0.0.1 or localhost) and connect as usual.
Second, the Console gives you secure and seamless access to resources inside your VPC. Authentication and authorization are evaluated via IAM before traffic reaches the VPC. The diagram below illustrates a user connecting to an AWS EIC.
As shown above, the EIC Endpoints Service provides a high degree of flexibility.
- It doesn’t require your VPC to have direct Internet connectivity
- No agent is needed on the resource(s) you wish to connect to
- It preserves existing workflows by allowing your preferred client software to connect
- IAM and Security Groups can be used to control access
4.2.2. Instance Connect Endpoint Security and Logging
IAM
The beauty of this solution lies in the fact that access can be limited to what the user needs: EIC Endpoints Service follows the important security requirements in terms of the separation of privileges for the control plane and the data plane. An administrator with full EC2 IAM privileges can create and control EIC Endpoints (the control plane). However, the same administrator cannot use those endpoints without also having EC2 Instance Connect IAM privileges (the data plane). The DevOps Engineers who may need to use EIC Endpoint to tunnel into VPC resources do not require control-plane privileges to do so.
CloudTrail
Records of data plane connections include the IAM principal making the request, their source IP address, the requested destination IP address, and the destination port. An example of connecting to the instance from the console:
{
“eventVersion”: “1.08”,
“userIdentity”: {
“type”: “FederatedUser”,
“principalId”: “123456789:Example”,
“arn”: “arn:aws:sts::123456789:ExampleUser”,
“accountId”: “123456789”,
“accessKeyId”: “ExampleKey”,
“sessionContext”: {
“sessionIssuer”: {
“type”: “IAMExample”,
“principalId”: “ExampleID”,
“arn”: “arn:aws:iam::123456789:user/exampleuser”,
“accountId”: “123456789”,
“userName”: “exampleuser”
},
“webIdFederationData”: {},
“attributes”: {
“creationDate”: “2023-07-24T12:24:55Z”,
“mfaAuthenticated”: “true”
}
}
},
“eventTime”: “2023-07-24T13:19:22Z”,
“eventSource”: “ec2-instance-connect.amazonaws.com”,
“eventName”: “SendSSHPublicKey”,
“awsRegion”: “eu-west-1”,
“sourceIPAddress”: “123456789”,
“userAgent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0”,
“requestParameters”: {
“instanceId”: “i-123456789”,
“instanceOSUser”: “ec2-user”,
“sSHPublicKey”: “ssh-ed25519Example\n”
},
“responseElements”: {
“requestId”: “b2a9eeae-7189-4b6c-ac8a-a41e56ea1a42”,
“success”: true
},
“requestID”: “b2a9eeae-7189-4b6c-ac8a-a41e56ea1a42”,
“eventID”: “161427f7-8f1e-459a-83f1-2f54aca879ca”,
“readOnly”: false,
“eventType”: “AwsApiCall”,
“managementEvent”: true,
“recipientAccountId”: “123456789”,
“eventCategory”: “Management”,
“tlsDetails”: {
“tlsVersion”: “TLSv1.3”,
“cipherSuite”: “TLS_AES_128_GCM_SHA256”,
“clientProvidedHostHeader”: “ec2-instance-connect.eu-west-1.amazonaws.com”
},
“sessionCredentialFromConsole”: “true”
}
4.3. Connection Examples
4.3.1. Via the Console
This gives you:
As you can see, the console does not login via SSM-user but via EC2-user and with a private IP.
4.3.2 Via the AWS CLI
To connect via the AWS CLI, make sure that you have the latest AWS CLI installed. Then you can use the new ec2-instance-connect ssh command from the AWS CLI. With this new command, AWS generates ephemeral keys for you to connect to your Instance.
Note that this command requires using the OpenSSH client and the latest version of the AWS CLI. You need IAM permissions, as detailed here, to use this command and connect. You can find an example below:
4.3.3 Via SSH Clients
Create your own private key:
ssh-keygen -t rsa -f mynew_key
Use the following AWS CLI command to authorize the user and push the public key to the Instance using the send-ssh-public-key command. To support this, you need the latest version of the AWS CLI.
aws ec2-instance-connect send-ssh-public-key –region eu-west-1 –instance-id i-0123456789example –availability-zone eu-west-1a –instance-os-user ec2-user –ssh-public-key file://mynew_key.pub
This gives back:
{
“RequestId”: “9c4bf3df-799d-4f40-9e2d-cbc3ed3bbe08”,
“Success”: true
}
After authentication, the public key is made available to the Instance through the Instance metadata for 60 seconds. During this time, connect to the Instance using the associated private key:
ssh ec2-user@[INSTANCE] \
-i [SSH-KEY] \
-o ProxyCommand=’aws ec2-instance-connect open-tunnel \
–instance-id %h’
This gives back:
With this method, plain SSH clients can still be used via short-lived keys.
4.3.4 Connecting to the Windows Machines
How to connect to Windows machines is similar:
aws ec2-instance-connect open-tunnel \
–instance-id i-0123456789example \
–remote-port 3389 \
–local-port any-port
In your RDP Client, set it up like this, mark that the –local-port is set to 5555
4.4 To Sum Up Instance Connect
As shown in the blog post above, Instance Connect allows you to connect to your private resources privately and securely without needing long-lived SSH keys and/or bastion hosts.
The drawback compared to AWS Sessions Manager is that you cannot log session logs towards Amazon S3 or Amazon CloudWatch. The benefit is that you don’t need to install custom agents on your machines, which makes this solution the preferred solution for older Instances and appliances running in your AWS environment.
5. Final Conclusion
AWS provides various methods to connect to Instances, including SSH, RDP, AWS Systems Manager Session Manager, EC2 Instance Connect and Endpoint, bastion hosts, and VPN. The choice of method depends on the operating system, security requirements, network configuration, and personal preference. By leveraging these connection methods, you can securely access and manage your AWS Instances based on your specific use cases and requirements.
What questions do you have after reading this blog post? We would love to answer them!
