Creating and managing secrets is perhaps as old as humans interacting with each other. Yet despite their critical importance, secrets remain one of the most vulnerable aspects of AWS infrastructure today. In our practice as Dev(Sec)Ops Engineers, we see these challenges daily accross our client environments.

To start, the question needs to be asked: “What exactly is a secret?”  In AWS and Cloud environments in general, a secret is anything used to access systems and/or data. These digital keys unlock potentially critical systems and must be protected from unauthorized access at all costs.

As Dev(Sec)Ops Engineers working in AWS, you are responsible for managing an extensive arry of secrets:

• RDS database usernames and passwords
• API keys for AWS services and third-party integrations
• CodeDeploy and deployment system credentials
• KMS encryption keys for data protection
• Private keys for SSL/TLS certificates and secure communications
• EC2 SSH keys and key pairs
• IAM user credentials for developers, QA, and operations teams
• Application service accounts and roles
• Any username/password combinations
• Sensitive configuration data that could aid attackers

The challenge is not just the volume … it’s the complexity of securing these secrets across multiple AWS accounts and other environments while maintaining operational efficiency.

The Hidden Dangers: How Secrets Get Compromised in AWS

 The Configuration File Trap

Storing secrets in configuration files is convenient but dangerous, especially in AWS environments where these files often end up in S3 buckets or EC2 instances. This risk was nicely illustrated in the RSA conference presentation “Red Team vs Blue Team on AWS” by Kolby Allen and Teri Radichel: Link to video

In the video Kolby deployed AWS resources using sample code found easily on the Internet.  Teri, simulating an attacker, conducted a penetration test and discovered a Lambda function’s configuration file stored in a S3 bucket (for convenience) complete with REDS database credentials in plain tekst.

If you must use configuration files in AWS:

• Maintain separate files for development, QA, and production AWS accounts
• NEVER commit these files to source control systems (GitHub, GitLab, Bitbucket)
• Use S3 bucket policies and encryption, but understand this is still suboptimal
• Consider AWS Systems Manager Parameter Store as a minimum improvement

This vulnerability is so important that it’s formally recognized in the MITRE attack framework:  [CWE-200: Exposure of Sensitive Information] (link)

Public Source Control: A Goldmine for AWS Account Takeovers

The most devastating secrets management failures occur when AWS credentials are committed to public repositories. The scale of this problem is staggering.

The people at TruffleHog constructed a live scanner for this:

https://forager.trufflesecurity.com/explore

This scanner continuously detects AWS access keys, secrets keys and other credentials in public Github commits, revealing the massive and constant stream of exposed AWS and other secrets.

This has a widespread impact: TruffleHog researchers discovered approximately 4,500 secrets among the top 1 million websites, many of which were AWS credentials, as detailed in their comprehensive analysis:

https://trufflesecurity.com/blog/4500-of-the-top-1-million-websites-leaked-source-code-secrets

Leaking these secrets can have far reaching financial consequences:

A Reddit user faced a 26.000 $ bill after IAM was compromised to execute crypto miners:

https://www.reddit.com/r/aws/comments/17p3v1e/account_got_hacked_and_get_26000k_bill/

A Developper was billed 14.000 $ on AWS following similar exposure:

https://dev.to/juanmanuelramallo/i-was-billed-for-14k-usd-on-amazon-web-services-17fn

AWS responded on these massive bills and is now actively scanning GitHub respositories through their AWS Credentials Exposed program and automatically disables discovery IAM access keys, but the frequency of these incidents remains alarmingly high as shown by the Trufflehog data.

Internal Systems: The False Security Blanket in AWS

Private repositories and internal systems create a dangerous illusion of security, even within AWS environments. The 2020 Twitter breach perfectly illustrates this vulnerability:

Attackers breached the perimeter, accessed internal Slack channels where developers had stored AWS credentials and other secrets, and used these to compromise infrastructure in a widely publicized incident:

https://www.zdnet.com/article/twitter-says-hackers-accessed-dms-for-36-users-in-last-weeks-hack

Secrets embedded in code even in fully private AWS services, proliferate across AWS services and do appear in:

• CloudWatch logs from Lambda functions and EC2 instances
• S3 bucket access logs
• CloudTrail event data
• Application Load Balancer logs
• Systems Manager Session Manager history

As such they do create additional attack vectors within your AWS environment.

Runtime Exposure: Secrets in AWS Production Workloads

Running AWS applications create numerous opportunities for secret exposure:

• Configuration files within EC2 instances or container images
• Environment variables visible in ECS task definitions or Lambda configurations
• Memory dumps from EC2 instances containing sensitive data
• Application caches in ElastiCache storing credentials
• CloudWatch logs revealing secrets in error messages
• EC2 instance metadata (IMDSv1) exposing IAM credentials
• Unencrypted S3 bucket metadata and tags
• AWS CloudShell command history
• Bash history on EC2 instances
• Container image layers in ECR with embedded secrets

This non-exhaustive list demonstrates how secrets can leak from multiple AWS vectors without proper handling.

What does AWS offer to fight this battle ? AWS Native Secrets Management

Adopt Just-in-Time Secret Retrieval with AWS Services

Core Principle: Store secrets in AWS-native management systems and retrieve them only when needed.

Instead of embedding secrets in configuration files, implement this AWS-secure workflow:

1. Store secrets in AWS Secrets Manager or Systems Manager Parameter Store
2. Retrieve secrets programmatically using AWS SDKs at runtime
3. Use IAM roles and policies for access control
4. Clear secrets from memory when no longer needed

AWS Access Control: Implement least-privilege IAM principles

• Developers cannot access production secrets via IAM policies
• Applications use IAM roles to retrieve only their required secrets
• Cross-account access is controlled via resource-based policies

AWS-Native Secrets Management Solutions

• AWS Systems Manager Parameter Store
• AWS Secrets Manager (Recommended for Production)

As an MSSP, we recommend AWS Secrets Manager as the gold standard for AWS environments.

Why Secrets Manager over Parameter Store?

• Enhanced security controls with fine-grained IAM integration
• Automatic secret rotation for RDS, DocumentDB, and Redshift
• Cross-account access capabilities essential for multi-account AWS strategies
• Comprehensive audit trails via CloudTrail integration
• No CloudFormation exposure risks unlike Parameter Store
• Encryption at rest using AWS KMS by default

As such we recommend the AWS Secrets Manager for Production workloads, the Encrypted Parameter Store can be used as a configuration store to fetch certain parameters.

How to use this in code ?

Python with boto3:

import boto3
import json

# Using IAM role-based authentication (recommended)
client = boto3.client('secretsmanager', region_name='us-east-1')

try:
    secret_response = client.get_secret_value(SecretId='prod/rds/mysql-credentials')
    secret_dict = json.loads(secret_response['SecretString'])
    
    # Use the secret
    db_username = secret_dict['username']
    db_password = secret_dict['password']
    
except ClientError as e:
    # Handle AWS-specific errors
    if e.response['Error']['Code'] == 'DecryptionFailureException':
        # Secrets Manager can't decrypt the protected secret text using the provided KMS key
        raise e
    elif e.response['Error']['Code'] == 'InternalServiceErrorException':
        # An error occurred on the server side
        raise e
    elif e.response['Error']['Code'] == 'InvalidParameterException':
        # Invalid parameter value
        raise e
    elif e.response['Error']['Code'] == 'InvalidRequestException':
        # Parameter value is not valid for the current state of the resource
        raise e
    elif e.response['Error']['Code'] == 'ResourceNotFoundException':
        # Can't find the resource that you asked for
        raise e

Via Infrastructure as Code:

• Cloudformation:
  • https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-secretsmanager-secret.html
• Terraform:
  • https://docs.aws.amazon.com/prescriptive-guidance/latest/secure-sensitive-data-secrets-manager-terraform/using-secrets-manager-and-terraform.html
• AWS CDK:
  • https://docs.aws.amazon.com/secretsmanager/latest/userguide/cdk.html
• Pulumi:
  • https://www.pulumi.com/registry/packages/aws/api-docs/secretsmanager/secret/

AWS Key Management Service (KMS)

AWS KMS provides dedicated encryption key management, solving the “where to store the encryption key” problem for AWS environments. KMS integrates seamlessly with Secrets Manager and most AWS services.

KMS Best Practices for Secrets:

• Use customer-managed KMS keys for production secrets
• Implement key rotation policies
• Use separate KMS keys per environment/account
• Leverage KMS key policies for fine-grained access control

AWS Systems Manager Parameter Store

While we recommend Secrets Manager for sensitive data, **Parameter Store** works well for:

• Non-sensitive configuration data
• Cost-conscious environments (free tier available)
• Simple use cases without rotation requirements

Multi-Account AWS Strategies

For AWS Organizations with multiple accounts (our recommended approach), consider:

Cross-Account Secrets Access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::PROD-ACCOUNT-ID:role/ApplicationRole"
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "secretsmanager:ResourceTag/Environment": "production"
        }
      }
    }
  ]
}

Multi-Region Considerations:

• Replicate critical secrets across AWS regions
• Use AWS Secrets Manager automatic replication
• Consider data residency requirements

AWS Implementation Best Practices from Our MSSP Experience

Security Architecture Considerations

• Design secure AWS deployment pipelines** using CodePipeline, CodeBuild, Github Actions with ci-cd plugins like trufflehog : https://undercodetesting.com/how-to-hunt-for-sensitive-credentials-using-trufflehog
• Implement comprehensive IAM access management with least-privilege principles
• Establish governance policies using AWS Config and AWS Organizations SCPs/RCPs
• Plan for secret rotation using AWS Secrets Manager automation
• Monitor and audit secret access via CloudTrail and CloudWatch

Operational Excellence in AWS

• Automate secret provisioning using AWS Lambda and CloudFormation/Terraform/…
• Implement emergency access procedures via AWS SSO and break-glass roles
• Establish incident response for compromised secrets using AWS Security Hub or other CSPM/CNAPP
• Regular security assessments using AWS Inspector and third-party tools
• Cost optimization by right-sizing Secrets Manager usage vs. Parameter Store

AWS-Specific Monitoring and Alerting

Set up CloudWatch alarms for:

• Unusual Secrets Manager API calls
• Failed secret retrievals
• Cross-account secret access
• KMS key usage anomalies

Conclusion

Effective AWS secrets management is not just about choosing the right AWS service it’s about implementing a comprehensive security strategy that leverages AWS-native capabilities while addressing the entire lifecycle of sensitive data. The examples and breaches discussed here represent real financial and reputational risks that AWS customers face daily.

Key AWS Takeaways:

1. Never store secrets in configuration files, S3 buckets, or source control
2. Use AWS Secrets Manager for production secrets management
3. Implement just-in-time secret retrieval with AWS SDKs
4. Apply least-privilege IAM policies and roles
5. Leverage AWS KMS for encryption key management
6. Plan for multi-account and multi-region secret strategies

The post AWS Secrets Management: Protecting Your Digital Keys in the Cloud appeared first on Cloudar.

Cloudar, an AWS Premier Consulting Partner in Belgium, is expanding beyond its borders after more than a decade. A new location in Utrecht will facilitate further growth in the Netherlands. We discuss Cloudar’s plans with Steyn Huizinga.

Steyn Huizinga, Managing Partner at Cloudar NL, is leading the company’s Dutch expansion efforts. With ten years of AWS expertise, he brings the necessary experience to develop this new market. Cloudar was founded in 2014 and has always had a strong focus on AWS, a defining characteristic that sets it apart from other cloud specialists in both Belgium and the Netherlands.

Smart in Belgium and the Netherlands

While Cloudar has primarily focused on Belgian customers in the past, it has always operated beyond the border, Huizinga explains. “After a delay due to COVID, now is the right time to open a dedicated Dutch office, allowing us to fully concentrate on the Dutch market.” According to Huizinga, it is important that Cloudar is not just seen as “those smart guys from Belgium,” but rather as “those smart guys from Belgium and the Netherlands.” The Belgian team will continue focusing on its home market, while the Dutch team will be dedicated to local customers.

The growth ambitions for the Netherlands are significant. “We aim to attract around ten new colleagues by the end of 2025, with plans for even stronger growth in the second year to meet demand,” says Huizinga. That demand is particularly high in Azure-dominated Netherlands, he notes.

AWS Premier Consulting Partner

Achieving AWS Premier Consulting Partner status is no small feat. With only a few hundred partners worldwide holding this title, Cloudar is part of an exclusive group. Maintaining this status requires complete dedication to AWS, with stringent certification requirements for a significant portion of the workforce. Cloudar successfully meets these requirements with a relatively compact team, allowing it to be mentioned alongside much larger market players.

Cloudar’s AWS expertise is demonstrated through six AWS Competencies and five Service Validations across various domains. Given the vastness of the AWS portfolio, excelling in all services is impossible. Instead, Cloudar specializes in five core areas: security, cloud-native development, containerization, FinOps, and AI technology.

A key part of Cloudar’s approach is balancing innovation with responsible cloud usage. Many organizations start enthusiastically with AWS but later face unexpectedly high costs or overlooked security requirements. Cloudar advocates a ‘job-zero’ approach, integrating security and FinOps from the outset to prevent unpleasant surprises down the line.

By focusing exclusively on AWS, Cloudar stays close to the source of innovation. The company gains early access to AWS roadmaps, allowing it to anticipate new developments. This position in the AWS ecosystem enables Cloudar to respond quickly to changes and advise customers on the most effective implementations. For example, a container workload can be deployed in seventeen different ways on AWS, and Cloudar helps determine the optimal approach for each specific case. However, AWS can still surprise everyone, so consultants must be well-versed in the existing offerings.

In 2024, Cloudar achieved the AWS Sovereign Cloud Competency, aligning with the growing need for cloud sovereignty. The company embraces the mantra “encrypt everything” to ensure maximum data security. Additionally, Cloudar leverages AWS to approach IoT data in a more human-centric way rather than solely relying on dashboards.

Different Customer Preferences

Huizinga notes clear differences in cloud preferences among customer segments: “ISVs often prefer to run their IT on AWS, while enterprise customers typically adopt a multicloud approach.” In such cases, Cloudar collaborates with Azure-focused colleagues, ensuring that Cloudar’s expertise remains dedicated to the AWS portion of a customer’s cloud strategy.

Choosing between cloud providers can be complex and should always be based on a clear cloud strategy, Huizinga emphasizes. “Organizations need to determine why they use Cloud X or Y for specific situations.” This is particularly important when internal IT operations run on Azure, while customer-facing platforms such as websites or retail environments operate on AWS. “You don’t want data to be transferred unnecessarily between different clouds.”

Dutch Customers Already on Board

For Cloudar, expanding into the Netherlands is a natural step in its growth journey. The company already serves several Dutch clients, including Coop, Mediahuis (operating in both Belgium and the Netherlands), and RGF, an HR service provider. “Our approach is focused on helping organizations leverage Amazon’s cutting-edge technologies for competitive advantage while maintaining the fundamental aspects of a solid cloud infrastructure,” says Huizinga.

Huizinga observes that many companies adopt a multicloud strategy, with Microsoft Azure commonly used for office applications. AWS is frequently the preferred choice for customer-facing platforms, portals, and digital initiatives. “Of course, this is the perspective we see from our AWS-focused standpoint,” he adds.

According to Huizinga, AWS’s ‘model freedom’ philosophy provides a key advantage, particularly in AI. “AWS aims to offer customers as many AI options as possible, giving organizations the flexibility to choose the best solution for their specific needs.”

The post Cloudar Reaches a New Milestone with Dutch Expansion appeared first on Cloudar.

What do you like about working for Cloudar? 

Cloudar’s management places great emphasis on personal development and the importance of respecting each individual’s unique identity. I am grateful to work for a company that values collaboration and innovation, and I look forward to seeing what the future holds for us. 

One of the reasons for our team’s success is our no-nonsense approach and get-things-done mentality, as well as our strong inter-team collaboration. Despite not possessing a technical background, I am in awe of the unbelievable level of technical expertise that exists within our team, which makes me feel privileged to be a part of it every day. If you were a fly on the wall in our workplace, you would witness all the intense and positive discussions aimed at answering particular questions or situations. Challenging each other keeps us sharp! 

What memory or anecdote will stick with you? 

At some point in life, everyone may experience a life-changing event that turns their world upside down. The empathy and humanity demonstrated within Cloudar are truly exceptional and something that I will never ever forget. 

What’s your most important work milestone so far? 

Although this is an opportunity to recall a personal milestone, I deliberately choose not to do so. Looking back to roughly two years ago and reflecting on what we have achieved together is truly remarkable. It is the unity among our people that drives our fantastic organization’s success every single day. This has been an unparalleled opportunity, and I remain immensely grateful for it. 

Want to be part of our dream team? Check out job opportunities!  

The post Employee in the Spotlight: Account Manager Raf Lenaerts appeared first on Cloudar.

#1 Operational Excellence

The operational excellence pillar focuses on running and monitoring systems. The goal is to continually improve processes and procedures for delivering business value. This involves automating changes, responding to events and defining standards to manage daily operations. The design principles for operational excellence in the cloud are:

• Perform operations as code: Define your entire workload as code and update it with code. By doing this you limit human error and enable consistent responses to events.
• Make small changes that are reversible: Design workloads to allow components to be updated regularly. Make changes in small increments that can be reversed if they don’t help resolve issues.
• Refine operations frequently: Keep looking for ways to improve operations procedures. Set up regular review moment to make sure procedures are effective and that the team is familiar with them.
• Anticipate failure: Identify potential sources of failure. Test your failure scenario and validate your understanding of their impact. Test workload and team responses to simulated events.
• Learn from all operational failures: Drive improvement by learning from all failures. Share what is learned across teams and the organization.

#2 Security

The security pilar focuses on protecting information and systems using risk assessment and mitigation. This pillar includes confidentiality and integrity of data, managing user permissions, and establishing controls to detect security events. These design principles can help you strengthen your workload security:

• Build a strong identity foundation: Implement the principle of least privilege and enforce separation of duties with appropriate authorizations. Centralize identity management.
• Create traceability: Monitor, alert and audit actions and changes to your environment in real time. Integrate log and metric collection to automatically investigate and take action.
• Automate security: Create secure architectures that are defined and managed as code in version-controlled templates.
• Protect data in transit and at rest: Classify your date into sensitivity levels and use appropriate mechanisms.
• Prepare for security events: Prepare for incidents by having incident management aligned with your organizational requirements. Run incidence response simulations and use automated tools.
• Keep people away from data: Reduce or eliminate the need for direct access or manual processing of data.

#3 Cost optimization

Cost optimization has a major impact on your bottom line. The cost optimization pillar helps remove all practices that lead to unnecessary costs or underutilized resources. It helps better understand spending over time and control fund allocation, selecting resources of the right type and quantity, and scaling to meet business needs without overspending. These are the design principles for cost optimization:

• Implement cloud financial management: Dedicate the necessary time and resources for building capability in this new domain of technology and usage management. Build capability through knowledge building, programs, resources, and processes to help you become a cost-efficient organization.
• Work out a consumption model: Pay only for the computing resources you consume, and increase or decrease usage depending on business requirements.
• Measure overall cost efficiency: Measure the business output of the workload and the costs associated with delivery. Use this data to understand the gains you make from increasing output, increasing functionality, and reducing cost.
• Don’t waste money on undifferentiated heavy lifting: Let AWS do the heavy lifting of data center operations and the operational burden of managing operation systems and applications with managed services.
• Analyze and attribute expenditure: With the help of the cloud, accurately identify the cost and usage of workloads. This allows for the transparent attribution of IT costs to revenue streams and workload owners and helps measure ROI.

#4 Reliability

The pillar of reliability includes practices that help workloads perform their intended functions and allow for faster recovery from failure. It covers distributed system design, recovery planning, and adapting to changing requirements. This helps companies avoid interruptions and improve availability.

These design principles can help you increase reliability:

• Test your recovery procedures: Test how your workload fail to validate your recovery procedures. Use automation to simulate failure scenarios and expose failure pathways.
• Increase aggregate system availability: Scale horizontally, replacing one large resource with multiple small resources to reduce the impact of a single failure.
• Don’t guess capacity: Avoid failure due to resource saturation by monitoring demand and workload utilization while automating resource levels.
• Manage change in automation: Make changes to your infrastructure using automation. Changes can then be tracked and reviewed.
• Automatically recover from failure: Trigger automation when a threshold is breached by monitoring for KPIs that measure business value. This allows for automatic notification and tracking of failures, and for automated recovery processes.

#5 Performance efficiency

The performance efficiency pillar focuses on efficiently allocating IT and computing resources. It includes selecting resource types and sizes optimized for workload requirements, monitoring performance, and maintaining efficiency as business needs evolve. These design principles can help you achieve and maintain efficient workloads in the cloud:

• Experiment often: Quickly carry out comparative testing using different types of instances, storage, or configuration with virtual and automatable resources.
• Go global in a snap: Deploy your workload in multiple AWS Regions, allowing you to provide lower latency and a better experience for your customers at minimal cost.
• Use serverless architectures: Remove the need to run and maintain physical servers, reducing operational burden and lowering transactional costs.
• Democratize advanced technologies: Make advanced technology implementation easier for your team by delegating complex tasks to your cloud vendor. Consider consuming the technology as a service.
• Consider mechanical sympathy: Use the technology approach that aligns best with your goals.

#6 Sustainability

The sustainability pillar seeks to minimize the environmental impacts of running cloud workloads. It includes using a shared responsibility model for sustainability, understanding impact, and maximizing utilization to minimize required resources and reduce downstream impacts. These design principles help maximize sustainability and minimize impact:

• Understand your impact: Measure the impact of your cloud workload and model the future impact of your workload. Include all sources of impact, compare the productive output with the total impact of your cloud workloads and use this data to establish
• Establish sustainability goals: For each cloud workload, establish long-term sustainability goals. Plan for growth, and architect your workloads so that growth results in reduced impact intensity measured against an appropriate unit, such as per user or per transaction.
• Maximize utilization: Right-size workloads and implement efficient design to ensure high utilization and maximize the energy efficiency of the underlying hardware
• Anticipate and adopt new, more efficient hardware and software offerings: Continually monitor and evaluate new, more efficient hardware and software offerings. Design for flexibility to allow for the rapid adoption of new efficient technologies.
• Use managed services: Maximize resource utilization by sharing services across a broad customer base. Use managed services that can help minimize your impact and adjust capacity to meet demand.
• Reduce the downstream impact of your cloud workloads: Reduce the amount of energy or resources required to use your services. Reduce or eliminate the need for customers to upgrade their devices to use your services. Test expected impact and test with customers to understand actual impact.

Ready to check off that list to get the most business value while making a positive impact? It’s quite a task! Of course, it’s impossible to do it all. It’s about making the choices that best suit your business. To identify what that is exactly, you can hire an expert partner like Cloudar.

Want to evaluate your company’s needs and design for the future? Reach out to us!

The post Check it: Aligning with the AWS Well Architected Framework appeared first on Cloudar.