Creating and managing secrets is perhaps as old as humans interacting with each other. Yet despite their critical importance, secrets remain one of the most vulnerable aspects of AWS infrastructure today. In our practice as Dev(Sec)Ops Engineers, we see these challenges daily accross our client environments.

To start, the question needs to be asked: “What exactly is a secret?”  In AWS and Cloud environments in general, a secret is anything used to access systems and/or data. These digital keys unlock potentially critical systems and must be protected from unauthorized access at all costs.

As Dev(Sec)Ops Engineers working in AWS, you are responsible for managing an extensive arry of secrets:

• RDS database usernames and passwords
• API keys for AWS services and third-party integrations
• CodeDeploy and deployment system credentials
• KMS encryption keys for data protection
• Private keys for SSL/TLS certificates and secure communications
• EC2 SSH keys and key pairs
• IAM user credentials for developers, QA, and operations teams
• Application service accounts and roles
• Any username/password combinations
• Sensitive configuration data that could aid attackers

The challenge is not just the volume … it’s the complexity of securing these secrets across multiple AWS accounts and other environments while maintaining operational efficiency.

The Hidden Dangers: How Secrets Get Compromised in AWS

 The Configuration File Trap

Storing secrets in configuration files is convenient but dangerous, especially in AWS environments where these files often end up in S3 buckets or EC2 instances. This risk was nicely illustrated in the RSA conference presentation “Red Team vs Blue Team on AWS” by Kolby Allen and Teri Radichel: Link to video

In the video Kolby deployed AWS resources using sample code found easily on the Internet.  Teri, simulating an attacker, conducted a penetration test and discovered a Lambda function’s configuration file stored in a S3 bucket (for convenience) complete with REDS database credentials in plain tekst.

If you must use configuration files in AWS:

• Maintain separate files for development, QA, and production AWS accounts
• NEVER commit these files to source control systems (GitHub, GitLab, Bitbucket)
• Use S3 bucket policies and encryption, but understand this is still suboptimal
• Consider AWS Systems Manager Parameter Store as a minimum improvement

This vulnerability is so important that it’s formally recognized in the MITRE attack framework:  [CWE-200: Exposure of Sensitive Information] (link)

Public Source Control: A Goldmine for AWS Account Takeovers

The most devastating secrets management failures occur when AWS credentials are committed to public repositories. The scale of this problem is staggering.

The people at TruffleHog constructed a live scanner for this:

https://forager.trufflesecurity.com/explore

This scanner continuously detects AWS access keys, secrets keys and other credentials in public Github commits, revealing the massive and constant stream of exposed AWS and other secrets.

This has a widespread impact: TruffleHog researchers discovered approximately 4,500 secrets among the top 1 million websites, many of which were AWS credentials, as detailed in their comprehensive analysis:

https://trufflesecurity.com/blog/4500-of-the-top-1-million-websites-leaked-source-code-secrets

Leaking these secrets can have far reaching financial consequences:

A Reddit user faced a 26.000 $ bill after IAM was compromised to execute crypto miners:

https://www.reddit.com/r/aws/comments/17p3v1e/account_got_hacked_and_get_26000k_bill/

A Developper was billed 14.000 $ on AWS following similar exposure:

https://dev.to/juanmanuelramallo/i-was-billed-for-14k-usd-on-amazon-web-services-17fn

AWS responded on these massive bills and is now actively scanning GitHub respositories through their AWS Credentials Exposed program and automatically disables discovery IAM access keys, but the frequency of these incidents remains alarmingly high as shown by the Trufflehog data.

Internal Systems: The False Security Blanket in AWS

Private repositories and internal systems create a dangerous illusion of security, even within AWS environments. The 2020 Twitter breach perfectly illustrates this vulnerability:

Attackers breached the perimeter, accessed internal Slack channels where developers had stored AWS credentials and other secrets, and used these to compromise infrastructure in a widely publicized incident:

https://www.zdnet.com/article/twitter-says-hackers-accessed-dms-for-36-users-in-last-weeks-hack

Secrets embedded in code even in fully private AWS services, proliferate across AWS services and do appear in:

• CloudWatch logs from Lambda functions and EC2 instances
• S3 bucket access logs
• CloudTrail event data
• Application Load Balancer logs
• Systems Manager Session Manager history

As such they do create additional attack vectors within your AWS environment.

Runtime Exposure: Secrets in AWS Production Workloads

Running AWS applications create numerous opportunities for secret exposure:

• Configuration files within EC2 instances or container images
• Environment variables visible in ECS task definitions or Lambda configurations
• Memory dumps from EC2 instances containing sensitive data
• Application caches in ElastiCache storing credentials
• CloudWatch logs revealing secrets in error messages
• EC2 instance metadata (IMDSv1) exposing IAM credentials
• Unencrypted S3 bucket metadata and tags
• AWS CloudShell command history
• Bash history on EC2 instances
• Container image layers in ECR with embedded secrets

This non-exhaustive list demonstrates how secrets can leak from multiple AWS vectors without proper handling.

What does AWS offer to fight this battle ? AWS Native Secrets Management

Adopt Just-in-Time Secret Retrieval with AWS Services

Core Principle: Store secrets in AWS-native management systems and retrieve them only when needed.

Instead of embedding secrets in configuration files, implement this AWS-secure workflow:

1. Store secrets in AWS Secrets Manager or Systems Manager Parameter Store
2. Retrieve secrets programmatically using AWS SDKs at runtime
3. Use IAM roles and policies for access control
4. Clear secrets from memory when no longer needed

AWS Access Control: Implement least-privilege IAM principles

• Developers cannot access production secrets via IAM policies
• Applications use IAM roles to retrieve only their required secrets
• Cross-account access is controlled via resource-based policies

AWS-Native Secrets Management Solutions

• AWS Systems Manager Parameter Store
• AWS Secrets Manager (Recommended for Production)

As an MSSP, we recommend AWS Secrets Manager as the gold standard for AWS environments.

Why Secrets Manager over Parameter Store?

• Enhanced security controls with fine-grained IAM integration
• Automatic secret rotation for RDS, DocumentDB, and Redshift
• Cross-account access capabilities essential for multi-account AWS strategies
• Comprehensive audit trails via CloudTrail integration
• No CloudFormation exposure risks unlike Parameter Store
• Encryption at rest using AWS KMS by default

As such we recommend the AWS Secrets Manager for Production workloads, the Encrypted Parameter Store can be used as a configuration store to fetch certain parameters.

How to use this in code ?

Python with boto3:

import boto3
import json

# Using IAM role-based authentication (recommended)
client = boto3.client('secretsmanager', region_name='us-east-1')

try:
    secret_response = client.get_secret_value(SecretId='prod/rds/mysql-credentials')
    secret_dict = json.loads(secret_response['SecretString'])
    
    # Use the secret
    db_username = secret_dict['username']
    db_password = secret_dict['password']
    
except ClientError as e:
    # Handle AWS-specific errors
    if e.response['Error']['Code'] == 'DecryptionFailureException':
        # Secrets Manager can't decrypt the protected secret text using the provided KMS key
        raise e
    elif e.response['Error']['Code'] == 'InternalServiceErrorException':
        # An error occurred on the server side
        raise e
    elif e.response['Error']['Code'] == 'InvalidParameterException':
        # Invalid parameter value
        raise e
    elif e.response['Error']['Code'] == 'InvalidRequestException':
        # Parameter value is not valid for the current state of the resource
        raise e
    elif e.response['Error']['Code'] == 'ResourceNotFoundException':
        # Can't find the resource that you asked for
        raise e

Via Infrastructure as Code:

• Cloudformation:
  • https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-secretsmanager-secret.html
• Terraform:
  • https://docs.aws.amazon.com/prescriptive-guidance/latest/secure-sensitive-data-secrets-manager-terraform/using-secrets-manager-and-terraform.html
• AWS CDK:
  • https://docs.aws.amazon.com/secretsmanager/latest/userguide/cdk.html
• Pulumi:
  • https://www.pulumi.com/registry/packages/aws/api-docs/secretsmanager/secret/

AWS Key Management Service (KMS)

AWS KMS provides dedicated encryption key management, solving the “where to store the encryption key” problem for AWS environments. KMS integrates seamlessly with Secrets Manager and most AWS services.

KMS Best Practices for Secrets:

• Use customer-managed KMS keys for production secrets
• Implement key rotation policies
• Use separate KMS keys per environment/account
• Leverage KMS key policies for fine-grained access control

AWS Systems Manager Parameter Store

While we recommend Secrets Manager for sensitive data, **Parameter Store** works well for:

• Non-sensitive configuration data
• Cost-conscious environments (free tier available)
• Simple use cases without rotation requirements

Multi-Account AWS Strategies

For AWS Organizations with multiple accounts (our recommended approach), consider:

Cross-Account Secrets Access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::PROD-ACCOUNT-ID:role/ApplicationRole"
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "secretsmanager:ResourceTag/Environment": "production"
        }
      }
    }
  ]
}

Multi-Region Considerations:

• Replicate critical secrets across AWS regions
• Use AWS Secrets Manager automatic replication
• Consider data residency requirements

AWS Implementation Best Practices from Our MSSP Experience

Security Architecture Considerations

• Design secure AWS deployment pipelines** using CodePipeline, CodeBuild, Github Actions with ci-cd plugins like trufflehog : https://undercodetesting.com/how-to-hunt-for-sensitive-credentials-using-trufflehog
• Implement comprehensive IAM access management with least-privilege principles
• Establish governance policies using AWS Config and AWS Organizations SCPs/RCPs
• Plan for secret rotation using AWS Secrets Manager automation
• Monitor and audit secret access via CloudTrail and CloudWatch

Operational Excellence in AWS

• Automate secret provisioning using AWS Lambda and CloudFormation/Terraform/…
• Implement emergency access procedures via AWS SSO and break-glass roles
• Establish incident response for compromised secrets using AWS Security Hub or other CSPM/CNAPP
• Regular security assessments using AWS Inspector and third-party tools
• Cost optimization by right-sizing Secrets Manager usage vs. Parameter Store

AWS-Specific Monitoring and Alerting

Set up CloudWatch alarms for:

• Unusual Secrets Manager API calls
• Failed secret retrievals
• Cross-account secret access
• KMS key usage anomalies

Conclusion

Effective AWS secrets management is not just about choosing the right AWS service it’s about implementing a comprehensive security strategy that leverages AWS-native capabilities while addressing the entire lifecycle of sensitive data. The examples and breaches discussed here represent real financial and reputational risks that AWS customers face daily.

Key AWS Takeaways:

1. Never store secrets in configuration files, S3 buckets, or source control
2. Use AWS Secrets Manager for production secrets management
3. Implement just-in-time secret retrieval with AWS SDKs
4. Apply least-privilege IAM policies and roles
5. Leverage AWS KMS for encryption key management
6. Plan for multi-account and multi-region secret strategies

The post AWS Secrets Management: Protecting Your Digital Keys in the Cloud appeared first on Cloudar.

In the dynamic world of cloud computing, effective and swift monitoring of your system’s health and performance is essential. The Amazon CloudWatch Agent stands out as a robust solution that empowers you to track custom metrics with ease. An important reason to use the agent, instead of just using CloudWatch agentless, is to collect additional metrics. A typical metric would be memory usage, but you can also monitor Windows WMI performance counters or Linux metrics from the /proc/stat file. While some organizations may lean towards specialized third-party monitoring solutions for their comprehensive features and dedicated support, the CloudWatch Agent presents a streamlined and cost-effective alternative, all within your AWS environment.

The CloudWatch agent simplifies the monitoring process and integrates seamlessly with your existing AWS ecosystem. However, before running the agent on any servers, it’s crucial to create a CloudWatch agent configuration file. Otherwise, no additional metrics beyond the default EC2 metrics that are being sent to CloudWatch will be monitored. Utilizing the provided CloudFormation template, you can initiate your monitoring capabilities without the complexities typically associated with setup and configuration.

We use CloudFormation because it automates and simplifies the process of managing related AWS resources, helping us to build and replicate infrastructure in a predictable and controlled manner.

Let’s look at how this solution works in detail.

• The template can be downloaded from the Cloudar GitHub repository here: cloudwatch.yaml

The CloudFormation template includes the following resources:

• SSMDocumentCloudWatchAgent: an SSM document that combines the AWS-ConfigureAWSPackage with the AmazonCloudWatchAgent and AmazonCloudWatch-ManageAgent to install and configure the CloudWatch agent.
• AgentConfigLinuxParameter and AgentConfigWinParameter: SSM parameters that store the configuration for Linux and Windows environments, respectively.
• CWAgentLinuxAssociation and CWAgentWinAssociation: SSM associations that ensure the execution of the CloudWatch agent installation and configuration.
• IAMPolicySSMParameter: an IAM policy that grants the necessary permissions for SSM and CloudWatch agent operations.
• IAMRoleforSSM and InstanceProfileforSSM: IAM role and instance profile that provide the required policies for SSM, CloudWatch agent, and access to the SSM parameters.

Deploying this template will create an SSM document, which is the result of merging two managed AWS SSM documents: AWS-ConfigureAWSPackage with AmazonCloudWatchAgent and AmazonCloudWatch-ManageAgent. This document will install the CloudWatch agent using the specified configuration.
The SSM associations will enforce the execution of this document to install and configure the CloudWatch agent. It will check the EC2 instances’ tags for the OS key (a dedicated OS tag is necessary for detecting the required configuration, an SSM association doesn’t explicitly detect OS details through e.g. metadata) and, depending on its value, will use either the Linux or Windows SSM parameter as its configuration. The created IAM resources are the preferred instance role, containing the necessary policies for SSM, the CloudWatch agent, and access to the SSM parameters that will be used.

The essential managed AWS policy for the CloudWatch agent is identified by the ARN:  arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

We are using SSM State Manager associations for better control, flexibility, and ease of management compared to manual installation or embedding in userdata fields.

The provided CloudWatch agent configurations are examples to help you get started quickly. We recommend you review and adapt them to better suit your requirements.

Once the CloudFormation template is deployed in your environment, the following process occurs:

• When an instance starts or is already running, SSM performs these steps:
  1. Discovery & Association Evaluation : SSM identifies the instance based on the OS tag, as specified in our two associations which contains the installation and CloudWatch agent configuration.
  2. Execution: If the instance meets the criteria, SSM executes the association document and installs the CloudWatch agent.
  3. Configuration Application: As the final step, the relevant agent configuration is applied. (Windows or Linux).

If everything proceeds as expected, CloudWatch will begin collecting custom metrics from your instances.

With the additional metrics now available in Amazon CloudWatch, you’ve taken a significant step toward monitoring and optimizing your environment. But don’t stop there! The next logical move is to implement CloudWatch alarms. These alarms allow you to proactively respond to critical events, ensuring the health and performance of your AWS resources. So go ahead, set up those alarms, and gain even more control over your AWS infrastructure! 🚀

The post Quickly Start Monitoring Your Workloads with the Amazon CloudWatch Agent appeared first on Cloudar.

Getting people to include Cloud financial management and FinOps principles in their daily operation of the cloud is what FinOps professionals like me have been striving for. With FinOps principles integrated more and more into standard cloud practice, an interesting question arises: should the FinOps engineer and the cloud engineer be two separate roles?

If we take DevOps as a precedent, we could argue that there is no way to do cloud engineering without being knowledgeable about DevOps principles and practices. The same could be said of FinOps: It is impossible to do quality cloud engineering without being aware of FinOps principles and best practices.

But wait, isn’t there a shared responsibility model for FinOps too? If we take that shared responsibility model, we can add the “cost efficiency in the cloud” to the responsibilities of the cloud engineers. That leaves us with the “cost efficiency of the cloud”, which surely means FinOps is here to stay? Yes and no. While the knowledge to properly ensure cost efficiency of the cloud is highly specific, the role filling up this layer of the FinOps shared responsibility model is akin to that of a program manager. Say hello to the FinOps Program Manager!

As FinOps becomes more of a tool in the toolbox of Cloud engineers rather than a separate practice (as should be the case), we need Program Managers to maintain and manage FinOps as a program within organizations. FinOps Program Managers are responsible for distributing knowledge, enabling managers and engineers in their access to the right information and making sure the organization has a clear way forward in developing FinOps.

In essence, FinOps becomes a knowledge space or practice utilized by cloud engineers and the program to ensure the cloud is adopted properly within the organization’s tech landscape. A dedicated FinOps profile is responsible for gathering and maintaining that knowledge. In practice, we often see this person and the FinOps Program Manager being one and the same. They are the driving force in gathering and maintaining cloud billing and cost optimization knowledge and supporting FinOps program management efforts within the organization. Based on their efforts, managers and cloud engineers can make informed decisions and have productive conversations on how to improve their cost efficiency in the cloud.

The bottom line: For FinOps to succeed in an organization, there is a strong need for a FinOps Program Manager who can lead up and down the chain of command and steer the course to keeping the shared responsibility model for FinOps in place.

Want some support to help your company get the most out of the cloud? Reach out to us!

The post Cloud cost optimization? Say hello to the FinOps Program Manager appeared first on Cloudar.

Sometimes we run into questions that seem simple, but that have a very complex answer. In this blog post we will explain why “Can I migrate between two Amazon Cognito User Pools” is one of those question, and why it usually leads to a new set of questions, like:

• Where do your users exist (do they federate from a different Identity Provider (IdP), or are they in the User Pool?)
• Are you using MFA in Cognito
• Which configuration settings can you change?
• Are you okay with a migration process in the application?
• Will all users sign in during a certain timeframe?
• Do you want to keep your endpoints and identifiers, or are new endpoints okay?

Lets look at why we are asking those questions, and which Cognito components are tricky to deal with.

Configuration

The configuration of a User Pool can be described by the API, or managed in an Infrastructure as Code tool. This makes it easy to recreate with the same settings. However, besides the new endpoints and ids (see below), you will have to update the federation settings (of you are using sign in through a third party).

Eg. If you are federating users from Active Directory (AD) using SAML, you will have to set up a new integration on the AD side as well, so that user can sign in via AD/SAML. This is the same process as setting up this IdP for the first time.

Endpoints and Ids

If you create a new user pool, it will get a new id. This means that the following things will change. Usually this is not a problem, as these are expected changes (if you migrate to a different product, similar things will change).

• The User Pool Id
• The App Client Ids and Secrets (you also create new App Clients)
• The domain used for the hosted UI. Technically you can move it, but this will include downtime. Using a new domain will always be easier (and give you more control about step-by-step updates and rollbacks)
• The User Pool Id and App Client Id are also part of the Tokens that Cognito returns. If you’re verifying them in your application (which you should), or are using a managed integration between an AWS service and Cognito (eg. Api Gateway), you will have to update these too.

Users and Groups

This is the difficult part: while there are APIs to list/describe users and groups, there is no way to export the password (hash) or the MFA (seed) of a user. This means that in the worst case scenario you have to disable MFA and all users will have to reset their password before they can log in again. There are a few solutions, but none of them are ideal.

Problem: You are using MFA in Cognito

This is the worst case scenario for a migration. There is no way to export or set MFA seeds in Cognito, and usually disabling MFA for everyone is not going to be an option

Problem: You are using the sub attribute in your application

The sub attribute is an unique identifier for every user, and migrating users will create a new identifier. To work around this, you could use a different (custom) attribute to safe an identifier in, and update your application to use that. Filling that attribute comes with its own complexities, though.

Option 1: You are using federation / sign in through a third party

You’re in luck! Since the user and their password exist in the other system, you can setup a new federation and users will still be able to sign in (but  make sure you’ve read the “Configuration” section above).

If you want to go this route to make migration (and failover) easier, there are third party IdPs that you can use via SAML/OIDC. Although in some cases it might make more sense to integrate directly with them (either in Cognito Identity Pools or in the application) instead of putting User Pools in between. One reason to keep Cognito, is that you can create multiple App Clients (and thus applications) on one pool (and this can be done with the IaC tool you are already using). Whereas if you integrate directly, you have to configure everything in the external service (and depending on the protocol you are using, this can be hard to debug)

If you are also using groups, you still have to export and recreate those. Take a look at the AWS Solution mentioned in option 2,  and its limitations for an example of doing that.

Option 2: You don’t mind resetting all passwords.

This is also doable, you can export all users, import them in the new pool and make them set a new password to sign in (if this is a limited set of internal users, you could also set a password for them, but you’d want them to pick their own either way)

There is even an AWS solution (read: Open Source software that you have to manage yourself) that can be used to do the export once or on a schedule (eg. to create backups). Make sure to also check the limitations.

Option 3: Migrating on first sign-in

You can add a Lambda Function to the new User Pool (there is example code, but you’d have to manage this yourself). This function will get the username and password, and can check the original pool and write the existing password to the new pool if the password is correct. Docs:

This has some downsides:

• The old User Pool needs to stay around until the migration is complete (you pay for active users, so this does not have to be a cost problem)
• This can take a long time, and potential never finish (if you have users that don’t sign in regularly)

Conclusion

The post Options for migrating between Amazon Cognito User Pools appeared first on Cloudar.

What is DevOps in Managed Services?

Let’s start with a quick review of what DevOps is. The word DevOps combines development and operations, reflecting the idea that it automates and integrate the processes between software development and IT teams. DevOps is a broad term for a combination of culture, practice and tools that accelerate an organization’s abilities to deliver applications and services. DevOps involves adjusting and reiteration products at fast pace. It is often visualized as a continuous loop, embodying the idea of automation and continuous improvement resulting from the seamless cooperation of developers and operators.

DevOps Managed Services helps enterprises implement changes to their digital products faster. It does this through continuous integration and continuous delivery or continuous deployment (CI/CD) tools, which helps monitor, update, and track updates. AWS gives you a set of flexible services that simplify provisioning and managing infrastructure, deploying application code, automating software release processes, and monitoring your application and infrastructure performance.

The DevOps advantage

The DevOps advantage comes down to speed while maintaining quality. The velocity of AWS DevOps helps organizations to better serve their customers and compete more effectively in the market by proving the following benefits:

• Speed: Get on the market faster and get ahead of competitors. With AWS services there is no infrastructure setup required or software to install, so you can focus on your core product.
• Reliability: Practices like continuous integration and continuous delivery test that each change is functional and safe. This lets you reliably deliver faster while ensuring the quality of updates and changes while keeping your users happy.
• Automation: AWS helps automate processes such as deployments, development & test workflows, container and configuration management. This also cuts down the budget for the deployment and support of IT infrastructure significantly.
• Scale: Automation and consistency help you manage changing systems more efficiently with reduced risk while scaling up.
• Flexibility: Making changes as you go instead of rebuilding your existing infrastructure from scratch lets businesses respond faster to customer and market needs.

AWS DevOps best practices

At Cloudar we put DevOps into practice, working with developers or application engineers throughout the entire project life-cycle. Together we decide on deployment strategies, application design optimization, CI/CD pipelines and monitoring requirements using both AWS native services and third party solutions. All developers have to do is write code, and the pipeline takes care of the rest.

We also provision, configure, and manage your AWS infrastructure resources using code and templates, known as infrastructure as code. This helps you manage your development, testing, and production environments in way that is more repeatable and efficient. We build templates monitoring and enforcing infrastructure compliance so clients can easily keep building and tearing down environments. Everything we set up adheres to AWS the AWS Well Architected Framework guidelines and can be fully managed in our Managed Services offering.

Ready to put DevOps into practice to accelerate your success? We enable your teams to focus on software while automating code deployments, infrastructure, security and systems management. Reach out to us!

The post The deal with DevOps: How AWS helps you deliver faster appeared first on Cloudar.

Indirection

With Terraform […] no intermediary sits between you and the service you’re controlling. Want an RDS instance? Terraform will make calls directly to the RDS API.

First of all, both Terraform and CloudFormation provide some level of indirection. CloudFormation converts a YAML or JSON template to a set of API calls. Terraform does the same, starting from an HCL template.

Having that transformation in the cloud can make it harder to see the underlying API responses, but that’s not because of the conversion between template and API. The same APIs are still getting called, and they will raise the same errors when there is something wrong with the requests. If you look in CloudTrail, you will see these requests together with the entire error message.

The way CloudFormation implements that translation layer (for newer resources) can add a second step after an API call fails. However, it also allows for standardized errors in other cases, like “Resource Already Exists” or “Only configurable when creating a resource”.

The prerequisite steps you have to take to use CloudFormation [StackSets] across multiple accounts also must be taken just to have CloudFormation spin up resources in multiple regions within the same account.

I’d argue that StackSets (and Service Catalog) are benefits of using CloudFormation. However, if they are too much overhead for a project, you can still deploy to multiple regions by adding a –region parameter to your deployment command. Although, by not using StackSets, you’d lose some convenience features, like being able to update multiple regions in parallel easily.

Logic

Terraform offers a rich set of data sources and transforming data with Terraform is a breeze.

I agree that CloudFormation could use more intrinsic functions, but that does not mean it’s severely lacking in functionality. In many cases, you can replace the missing functions by being stricter about the acceptable inputs.

Speed

AWS CloudFormation is S-L-O-W.

CloudFormation can seem slow because it tries very hard not to get into a state where your infrastructure is broken. Both CloudFormation and Terraform try to execute as much in parallel as possible (keeping dependencies between resources in mind). However, CloudFormation will not start deleting resources until every creation and update has been completed. On top of that, CloudFormation will validate that all actions (deletes included) finish before it considers the deployment complete.

Sync vs Async

Wrapping CI/CD tooling around AWS CloudFormation is […] a difficult matter of polling stack updates until the right stack state bubbles up from an aws cloudformation CLI query.

Here Greg argues that Terraform is better because polling for changes can be challenging to do. However, if you’re okay with executing a command-line tool (like Terraform), you can AWS CloudFormation deploy, and the AWS CLI will do all that for you.

Even without the CLI, running async comes with some significant advantages:

• You don’t have to worry about the build tool (or your laptop) crashing or losing internet connectivity
• Everyone with access to the AWS account can see when an update is in progress and poll independently for changes
• You don’t need a process that keeps running during the change. The deployment will keep running until it’s done.

Running 100% as a managed service also includes the following advantages:

• You don’t need to execute any code on your machine
• You don’t need to worry about different tool versions and upgrades between them
• You don’t need an access key with full admin permissions
• You get state management included by default (instead of having to configure it yourself).

That safer approach has the potential of doubling the time it takes to execute a change (assuming that a create/update takes as long as a delete), but having automatic rollbacks is frequently worth the extra time.

Portability

If you learn AWS CloudFormation, then guess what: you can’t take your skills with you. If you put forth the effort to learn Terraform, then you can take your skills to any other cloud.

While the workflow might stay the same, the resources that you create are going to be vastly different between clouds. The primary skill that you should learn (independently from the tool) is how AWS works. This will make your skills transferable between tools instead of between clouds. You wouldn’t expect carpenters to list their experience with a hammer (even if it’s useful for all kinds of projects): instead you want to know what they can build.

Importing existing resources

I can’t even begin to tell you how to import resources with AWS CloudFormation, and neither can AWS’s engineers.

This is pure hyperbole. Importing resources into CloudFormation is documented, and the console has a basic wizard.

If we ignore the exaggeration, there is indeed still room for improvement. While all new resource types should support importing, there is still a long tail of old resources that are not yet supported. “All resources should support resource import” is a valid point. However, CloudFormation was usable before the introduction of this feature. Whether this is a dealbreaker will depend on how frequently you expect to import unsupported resources.

Service Quotas

The list of quotas for the AWS CloudFormation service is just hilarious.

I don’t think that documented limitations are necessarily a bad thing, and I wonder how many of those quotas people actually run into. It’s been a long time since I had to look at that page.

TL;DR

CloudFormation makes different trade-offs than Terraform. Looking at its sharp edges without considering the advantages they bring is unfair.

Use the tool that matches your requirements. If you want safe deployments, tight integration within AWS, and minimal operational overhead, CloudFormation will probably fit the bill.

The post #CloudartotheRescue: DO use AWS CloudFormation (a response) appeared first on Cloudar.

From fuel gauge to disk space alert

Let’s use this trip down memory lane as a stepping stone. Imagine your car is a simple EC2 instance. How would you feel if you had to log onto your server and open up Explorer or run a df just to find out how much disk space you had left on your system drive? What would you think of having to check your dashboards every day? Would you be content with a virtual dipstick, as it were? 

Thankfully, this is not our reality today. Even traditional monitoring systems have long been capable of providing threshold-based alerts. Think of them as configurable low fuel indicator alerts for your systems.

Threshold-based alerts: a solid start

The question we need to ask ourselves is whether we are content with threshold-based alerts. A single gigabyte of free disk space might be plenty on one server and impending doom on the next. How do we fix this? Percentage-based thresholds are an interesting option, but they also come with a few caveats. While 5% might be ample free space for a 20TB file server, it would be problematic for a build server.

So you need to differentiate. And guess. Can you predict your system needs based on historical events and best practices?

Now, if there is one thing computers are better at than humans, it’s spotting patterns. They can learn from those patterns and deduce when things are “not normal”. Then, they will kindly alert you when there is cause for alarm. Much like the “miles to empty” feature in your car.

Putting machine learning to good use

And that is exactly what AWS DevOps Guru will do for you. The service uses machine learning techniques to spot anomalies in your environment and alert you before accidents happen. DevOps Guru uses machine learning models that were nurtured by years of Amazon.com and AWS operational excellence. You can use these models to identify anomalous application behavior (e.g. increased latency, error rates, resource constraints, etc.) and surface critical issues that could cause potential outages or service disruptions.

As an official AWS Managed Service Provider, Cloudar is constantly looking for ways to improve customer service. We help our customers to reach the highest levels of uptime. Or more specifically: the lowest levels of unplanned downtime, which is different, but I digress.

And while we still use traditional threshold-based monitoring tools (which do a great job in their own respect), using ML-based predictive monitoring and anomaly detection is something we have been doing for a while. With, I must admit, mixed feelings…

The predictive monitoring trap

Let’s circle back to our story about fuel. Imagine your car yelling at you to stop for gas every few hundred miles, only for you to arrive at the gas station and learn that the tank is half full. That is exactly the issue with a lot of tools that provide anomaly detection: false positives. 

At first glance, false positives don’t seem too bad. But trust me, they are. I would even argue that they can be as bad as missing a true alert. Any on-call engineer will tell you what happens when you receive too many false positives: you stop paying attention.

So yes, there is a lot of complexity in building an ML-based monitoring tool. And our experience as a launch partner (bragging rights here) is that DevOps Guru does a good job when it comes to limiting false positives. But of course, we are looking forward to seeing the tool grow even more.

Did my blog post spark your interest? Cloudar is hosting an EMEA DevOps Immersion Day on June 22 with a strong focus on DevOps Guru. The Immersion Days in the US were a big success and we are organizing this EMEA session with AWS to accommodate EMEA timezones. Sign up here.

The post How machine learning can drastically limit your downtime appeared first on Cloudar.

Installation

To use this new feature, you need the latest version of the CloudFormation cli. You can install this with pip install cloudformation-cli (tip: you can use pipx to keep your python environment clean). If you already had an older version installed, run pip install --upgrade cloudformation-cli

Starting a new project

To create a new module, we follow a similar process as with Resource Providers. For this post, we’re going to create a module to set up an S3 Bucket and a CloudFront Distribution for static website hosting.

We first create a new directory, after that we run cfn init. This will ask us for input, and we can specify that we want to create a module for our generic static website

~$ cfn init
Initializing new project
Do you want to develop a new resource(r) or a module(m)?.
>> m
What is the name of your module type?
(<Organization>::<Service>::<Name>::MODULE)
>> Cloudar::Generic::StaticWebsite::MODULE
Directory /cloudar_generic_staticwebsite/fragments Created
Initialized a new project in /cloudar_generic_staticwebsite

Writing our module

We already had a generic CloudFormation template that was doing a similar thing. So we deleted sample.json and replaced it with our own website.yaml. Officially, only json is supported, but in our test yaml worked fine. Since this is a test, we’re okay with taking the unsupported path. For use in a production environment, we can use cfn-flip to convert our template to json.

Depending on your template, you might not have to change anything. Going over the resources to make sure they have clear names and running cfn-lint isn’t a bad idea though. You can see the template we created on GitHub.

Deploying our module

We already had our AWS credentials configured for the AWS cli. Which meant that the only thing we had to do for our deployment was running cfn submit --region us-east-1. If you are submitting multiple times, you can add --set-default to change the default version after submitting it.

You can see the deployed version, with all its properties in the CloudFormation console. This is also a great place to see all the resources our module will create

Using our module

This is where it gets really interesting, we are going to use the module we created. We made a template with one resource (see Github for the full template):

Resources:
  StaticWebsite:
    Type: "Cloudar::Generic::StaticWebsite::MODULE"
    Properties:
      DomainName: !Ref DomainName
      HostedZoneId: !Ref HostedZoneId

CloudFormation will expand that for us, and put every resource that we defined in our module in our processed template.

After waiting for our deployment to finish, we can see all created resources and both our original and the processed template in the CloudFormation console.

Next steps

We’re very excited about this new release and are looking forward to seeing where this might lead. We also have a list of things we think might improve this even more (like an AWS::ModuleName pseudo-parameter), and will probably start conversations about that on the roadmap or issue tracker soon.

The post AWS CloudFormation Modules: A first look appeared first on Cloudar.

This blog post will highlight something you might not know yet in four different domains: writing templates, securing deployments, usability, and extensibility. Hopefully, this allows you to do even more with CloudFormation

CloudFormation Linting

One of the best ways to increase your CloudFormation productivity is to minimize the time you’re waiting on a stack that failed to deploy. A great way to do that is to use cfn-lint to make sure you don’t start the deploy if there are errors in your template.

Other tools like cfn-nag, cfn-guard, or taskcat will help you write secure, compliant, and region-independent templates, respectively. Cfn-lint is the one that will save you the most time when authoring templates by flagging misconfigurations, invalid properties, and misformatted templates. It also is easy to integrate into your IDE, pre-commit, or a CI/CD pipeline.

If you’re already using it, I don’t need to convince you of its value. I do, however, have some tips for getting the most out of it.

• You can include experimental rules with --include-experimental.
• You can add your own checks (written in python) with --append-rules
• You can enable more built-in rules with --include-checks I. The I stands for info. One of these informational checks that I find very useful is a warning if a resource that might contain state has no UpdateReplacePolicy/DeletionPolicy.

Another cool feature is that the linter can create a resource graph from your template. This is used by the tool to indicate when you write a circular dependency. It’s just as useful to resolve those circles or create a visual representation of your template.

IAM Integration

There are also some features of CloudFormation itself that I want to highlight. The first is how you can leverage its integration with IAM to prevent people from skipping the infrastructure-as-code step without completely locking down the account.

One way to accomplish that is to use a CloudFormation Service Role tied to your stack. This way, you can give minimal permission to your users and more broad permissions to CloudFormation. If you’re using ServiceCatalog to deploy infrastructure, you’re probably doing something very similar. This solution comes with some IAM complexity. The person creating the stack needs the iam:PassRole permission, but once set up, everyone who can update the stack will do so with that Role assumed by CloudFormation (whether you want that depends on your use case).

If you do not want to think about the effects of iam:PassRole, you can use the aws:CalledVia IAM condition in the IAM policy of the person using CloudFormation. Being an IAM condition comes with its own set of challenges, but you might be more familiar with those.

You use this method by specifying that certain actions are only allowed if the aws:CalledVia key is present and set to CloudFormation, and IAM will deny every request going directly to the service. The documentation has a helpful diagram
.

Parameter Validation

When using CloudFormation to deploy infrastructure, you can make it easier to put in the right parameters by using AllowedPattern and AllowedValues. Both of these only consider one parameter at a time. There is a way to validate Parameters that have a relationship with other parameters, though. Because you might want to validate that two parameters are mutually exclusive or that if one parameter is set, another parameter also has to have a value.

This is probably the most hidden CloudFormation feature because it’s documented in the Service Catalog documentation. And even though the Rules section seems to be a Service Catalog feature, you can use it directly in CloudFormation too. I wrote more about that in a previous blog post.

Resource Providers

And this blogpost would not be complete without mentioning resource providers. Giving a full example of writing one would be its own post. Still, I’d like to show how this successor to custom resources has a lot of potential.

Resource Providers work similar to Custom Resources, and you will not see a big difference in your templates. They should be easier to manage, though (once they can be deployed with CloudFormation – the CloudFormation team is actively working on that). When you write your own Resource Provider, you also implement and expose a Read and List handler. This new approach gives you almost the same capabilities as a native CloudFormation resource.

By creating resource providers, modern features like drift detection and resource import work with your own resources – and will work with every new resource that AWS releases support for.

There is also an integration with AWS Config for your Resource Providers, and AWS is developing more and more resource providers in the open.

Go build

We looked at 4 different ways to improve your CloudFormation experience, and I’m looking forward to hearing your favorite tips and tricks. Good luck making things!

The post Level up your CloudFormation usage with these 4 tips appeared first on Cloudar.

The post Infrastructure as Code : CloudFormation in practice (webinar recording) appeared first on Cloudar.