AWS Secrets Management: Protecting Your Digital Keys in the Cloud

Introduction

Creating and managing secrets is perhaps as old as humans interacting with each other. Yet despite their critical importance, secrets remain one of the most vulnerable aspects of AWS infrastructure today. In our practice as Dev(Sec)Ops Engineers, we see these challenges daily accross our client environments.

To start, the question needs to be asked: “What exactly is a secret?”  In AWS and Cloud environments in general, a secret is anything used to access systems and/or data. These digital keys unlock potentially critical systems and must be protected from unauthorized access at all costs.

 

As Dev(Sec)Ops Engineers working in AWS, you are responsible for managing an extensive arry of secrets:

 

• RDS database usernames and passwords
• API keys for AWS services and third-party integrations
• CodeDeploy and deployment system credentials
• KMS encryption keys for data protection
• Private keys for SSL/TLS certificates and secure communications
• EC2 SSH keys and key pairs
• IAM user credentials for developers, QA, and operations teams
• Application service accounts and roles
• Any username/password combinations
• Sensitive configuration data that could aid attackers

 

The challenge is not just the volume … it’s the complexity of securing these secrets across multiple AWS accounts and other environments while maintaining operational efficiency.

 

The Hidden Dangers: How Secrets Get Compromised in AWS

 The Configuration File Trap

Storing secrets in configuration files is convenient but dangerous, especially in AWS environments where these files often end up in S3 buckets or EC2 instances. This risk was nicely illustrated in the RSA conference presentation “Red Team vs Blue Team on AWS” by Kolby Allen and Teri Radichel: Link to video

In the video Kolby deployed AWS resources using sample code found easily on the Internet.  Teri, simulating an attacker, conducted a penetration test and discovered a Lambda function’s configuration file stored in a S3 bucket (for convenience) complete with REDS database credentials in plain tekst.

 

If you must use configuration files in AWS:

• Maintain separate files for development, QA, and production AWS accounts
• NEVER commit these files to source control systems (GitHub, GitLab, Bitbucket)
• Use S3 bucket policies and encryption, but understand this is still suboptimal
• Consider AWS Systems Manager Parameter Store as a minimum improvement

 

This vulnerability is so important that it’s formally recognized in the MITRE attack framework:  [CWE-200: Exposure of Sensitive Information] (link)

 

Public Source Control: A Goldmine for AWS Account Takeovers

The most devastating secrets management failures occur when AWS credentials are committed to public repositories. The scale of this problem is staggering.

The people at TruffleHog constructed a live scanner for this:

https://forager.trufflesecurity.com/explore

This scanner continuously detects AWS access keys, secrets keys and other credentials in public Github commits, revealing the massive and constant stream of exposed AWS and other secrets.

 

This has a widespread impact: TruffleHog researchers discovered approximately 4,500 secrets among the top 1 million websites, many of which were AWS credentials, as detailed in their comprehensive analysis:

https://trufflesecurity.com/blog/4500-of-the-top-1-million-websites-leaked-source-code-secrets

 

Leaking these secrets can have far reaching financial consequences:

 

A Reddit user faced a 26.000 $ bill after IAM was compromised to execute crypto miners:

https://www.reddit.com/r/aws/comments/17p3v1e/account_got_hacked_and_get_26000k_bill/

 

A Developper was billed 14.000 $ on AWS following similar exposure:

https://dev.to/juanmanuelramallo/i-was-billed-for-14k-usd-on-amazon-web-services-17fn

 

AWS responded on these massive bills and is now actively scanning GitHub respositories through their AWS Credentials Exposed program and automatically disables discovery IAM access keys, but the frequency of these incidents remains alarmingly high as shown by the Trufflehog data.

 

Internal Systems: The False Security Blanket in AWS

Private repositories and internal systems create a dangerous illusion of security, even within AWS environments. The 2020 Twitter breach perfectly illustrates this vulnerability:

 

Attackers breached the perimeter, accessed internal Slack channels where developers had stored AWS credentials and other secrets, and used these to compromise infrastructure in a widely publicized incident:

https://www.zdnet.com/article/twitter-says-hackers-accessed-dms-for-36-users-in-last-weeks-hack

 

Secrets embedded in code even in fully private AWS services, proliferate across AWS services and do appear in:

• CloudWatch logs from Lambda functions and EC2 instances
• S3 bucket access logs
• CloudTrail event data
• Application Load Balancer logs
• Systems Manager Session Manager history

As such they do create additional attack vectors within your AWS environment.

 

Runtime Exposure: Secrets in AWS Production Workloads

Running AWS applications create numerous opportunities for secret exposure:

• Configuration files within EC2 instances or container images
• Environment variables visible in ECS task definitions or Lambda configurations
• Memory dumps from EC2 instances containing sensitive data
• Application caches in ElastiCache storing credentials
• CloudWatch logs revealing secrets in error messages
• EC2 instance metadata (IMDSv1) exposing IAM credentials
• Unencrypted S3 bucket metadata and tags
• AWS CloudShell command history
• Bash history on EC2 instances
• Container image layers in ECR with embedded secrets

This non-exhaustive list demonstrates how secrets can leak from multiple AWS vectors without proper handling.

 

What does AWS offer to fight this battle ? AWS Native Secrets Management

Adopt Just-in-Time Secret Retrieval with AWS Services

Core Principle: Store secrets in AWS-native management systems and retrieve them only when needed.

Instead of embedding secrets in configuration files, implement this AWS-secure workflow:

1. Store secrets in AWS Secrets Manager or Systems Manager Parameter Store
2. Retrieve secrets programmatically using AWS SDKs at runtime
3. Use IAM roles and policies for access control
4. Clear secrets from memory when no longer needed

 

AWS Access Control: Implement least-privilege IAM principles

• Developers cannot access production secrets via IAM policies
• Applications use IAM roles to retrieve only their required secrets
• Cross-account access is controlled via resource-based policies

 

AWS-Native Secrets Management Solutions

• AWS Systems Manager Parameter Store
• AWS Secrets Manager (Recommended for Production)

 

As an MSSP, we recommend AWS Secrets Manager as the gold standard for AWS environments.

 

Why Secrets Manager over Parameter Store?

• Enhanced security controls with fine-grained IAM integration
• Automatic secret rotation for RDS, DocumentDB, and Redshift
• Cross-account access capabilities essential for multi-account AWS strategies
• Comprehensive audit trails via CloudTrail integration
• No CloudFormation exposure risks unlike Parameter Store
• Encryption at rest using AWS KMS by default

As such we recommend the AWS Secrets Manager for Production workloads, the Encrypted Parameter Store can be used as a configuration store to fetch certain parameters.

 

How to use this in code ?

 

Python with boto3:

 

import boto3
import json

# Using IAM role-based authentication (recommended)
client = boto3.client('secretsmanager', region_name='us-east-1')

try:
    secret_response = client.get_secret_value(SecretId='prod/rds/mysql-credentials')
    secret_dict = json.loads(secret_response['SecretString'])
    
    # Use the secret
    db_username = secret_dict['username']
    db_password = secret_dict['password']
    
except ClientError as e:
    # Handle AWS-specific errors
    if e.response['Error']['Code'] == 'DecryptionFailureException':
        # Secrets Manager can't decrypt the protected secret text using the provided KMS key
        raise e
    elif e.response['Error']['Code'] == 'InternalServiceErrorException':
        # An error occurred on the server side
        raise e
    elif e.response['Error']['Code'] == 'InvalidParameterException':
        # Invalid parameter value
        raise e
    elif e.response['Error']['Code'] == 'InvalidRequestException':
        # Parameter value is not valid for the current state of the resource
        raise e
    elif e.response['Error']['Code'] == 'ResourceNotFoundException':
        # Can't find the resource that you asked for
        raise e

 

 

Via Infrastructure as Code:

• Cloudformation:
  • https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-secretsmanager-secret.html
• Terraform:
  • https://docs.aws.amazon.com/prescriptive-guidance/latest/secure-sensitive-data-secrets-manager-terraform/using-secrets-manager-and-terraform.html
• AWS CDK:
  • https://docs.aws.amazon.com/secretsmanager/latest/userguide/cdk.html
• Pulumi:
  • https://www.pulumi.com/registry/packages/aws/api-docs/secretsmanager/secret/

 

AWS Key Management Service (KMS)

AWS KMS provides dedicated encryption key management, solving the “where to store the encryption key” problem for AWS environments. KMS integrates seamlessly with Secrets Manager and most AWS services.

 

KMS Best Practices for Secrets:

• Use customer-managed KMS keys for production secrets
• Implement key rotation policies
• Use separate KMS keys per environment/account
• Leverage KMS key policies for fine-grained access control

 

AWS Systems Manager Parameter Store

While we recommend Secrets Manager for sensitive data, **Parameter Store** works well for:

• Non-sensitive configuration data
• Cost-conscious environments (free tier available)
• Simple use cases without rotation requirements

 

Multi-Account AWS Strategies

For AWS Organizations with multiple accounts (our recommended approach), consider:

Cross-Account Secrets Access:

 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::PROD-ACCOUNT-ID:role/ApplicationRole"
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "secretsmanager:ResourceTag/Environment": "production"
        }
      }
    }
  ]
}

 

Multi-Region Considerations:

• Replicate critical secrets across AWS regions
• Use AWS Secrets Manager automatic replication
• Consider data residency requirements

 

AWS Implementation Best Practices from Our MSSP Experience

Security Architecture Considerations

• Design secure AWS deployment pipelines** using CodePipeline, CodeBuild, Github Actions with ci-cd plugins like trufflehog : https://undercodetesting.com/how-to-hunt-for-sensitive-credentials-using-trufflehog
• Implement comprehensive IAM access management with least-privilege principles
• Establish governance policies using AWS Config and AWS Organizations SCPs/RCPs
• Plan for secret rotation using AWS Secrets Manager automation
• Monitor and audit secret access via CloudTrail and CloudWatch

 

Operational Excellence in AWS

• Automate secret provisioning using AWS Lambda and CloudFormation/Terraform/…
• Implement emergency access procedures via AWS SSO and break-glass roles
• Establish incident response for compromised secrets using AWS Security Hub or other CSPM/CNAPP
• Regular security assessments using AWS Inspector and third-party tools
• Cost optimization by right-sizing Secrets Manager usage vs. Parameter Store

 

AWS-Specific Monitoring and Alerting

Set up CloudWatch alarms for:

• Unusual Secrets Manager API calls
• Failed secret retrievals
• Cross-account secret access
• KMS key usage anomalies

 

Conclusion

Effective AWS secrets management is not just about choosing the right AWS service it’s about implementing a comprehensive security strategy that leverages AWS-native capabilities while addressing the entire lifecycle of sensitive data. The examples and breaches discussed here represent real financial and reputational risks that AWS customers face daily.

 

Key AWS Takeaways:

1. Never store secrets in configuration files, S3 buckets, or source control
2. Use AWS Secrets Manager for production secrets management
3. Implement just-in-time secret retrieval with AWS SDKs
4. Apply least-privilege IAM policies and roles
5. Leverage AWS KMS for encryption key management
6. Plan for multi-account and multi-region secret strategies