The first one was filed on November 19, 2017. Seven years ago. The ask: let an admin change a customer’s email address in Jira Service Management. Not migrate accounts through a four-step workaround involving Atlassian ID. Just… change an email address. The kind of thing you’d expect a junior developer to ship on a Tuesday afternoon. As of today, it has 779 votes, 447 watchers, and a status of “Future Consideration.” Which, in SaaS-speak, translates roughly to: “we heard you, we filed it, please stop asking.”

The second one landed in January 2019. The request: make Confluence’s alphabetical page sorting persistent, so new pages don’t just pile up at the bottom like uninvited guests. Six years later: 600 votes, 261 watchers, status “Under Consideration.” Progress!

Now, to be fair to Atlassian, they’re not uniquely terrible. Every major SaaS vendor has a graveyard of feature requests exactly like these. Sensible, obvious, clearly wanted by thousands of paying customers. Just… never quite prioritized. Because they have a roadmap. And you’re not on it.

This is the part of the SaaS brochure they don’t show you.

The Pitch vs. The Reality

SaaS vendors are exceptionally good at one thing before you sign: making you feel like you’re about to get exactly what you need. The demos are polished. The slide decks are beautiful. The onboarding is smooth. Eighty percent of your requirements? Covered, on day one.

It’s that remaining twenty percent where things get interesting.

That twenty percent is where your actual workflows live. The edge cases. The things specific to how your organization actually operates. And when you file a support ticket asking about them, you enter a fascinating parallel universe where time moves differently. Features are “on the roadmap.” Updates will come “in a future release.” Your vote has been registered. Thank you for your feedback.

Meanwhile, you’re paying. Every month. For the product as it exists, not as it was promised.

The Numbers Are Telling

The scale of SaaS sprawl is difficult to overstate. According to BetterCloud’s annual State of SaaSOps report, the average number of SaaS applications per company peaked at 130 in 2022 and even after a wave of consolidation, still sits at over 100 today. That’s more than 100 subscriptions to manage, renew, secure, and integrate. For every single organization.

The waste embedded in that sprawl is just as striking. Gartner estimates that approximately 30% of purchased SaaS licenses go unused, what they bluntly call “toxic spend.” BetterCloud puts a price tag on it: companies report wasting an average of more than $135,000 per year on unused software licenses alone. And Gartner projects SaaS spending will continue growing at around 19% annually, reliably outpacing the budgets meant to fund it.

Do the arithmetic on a typical mid-market tool. Fifty users. €50 per user per month. That’s €30,000 per year. Every year. With annual price increases that arrive in your renewal email as a polite fait accompli. After five years, you’ve spent €150,000-plus on software you don’t own, can’t modify, and can’t easily leave and somewhere between a quarter and a third of those licenses have been sitting idle.

Something Changed

Here’s what’s different in 2026: building software got dramatically cheaper and faster. Not incrementally but by an order of magnitude.

Tools like Claude Code and Kiro represent a new category of agentic coding assistants. They don’t just suggest the next line. They can take a requirement, reason through a solution, write the code, test it, catch errors, and iterate, with minimal human supervision. What previously required a team of developers and months of work can now be done by one technically capable person in days.

This isn’t science fiction. It’s happening right now in engineering teams across the world.

And it fundamentally changes the math on one of the oldest questions in IT: should we build or buy?

What You Get When You Build Your Own

When you build a custom application -even a relatively simple internal tool- you build exactly what you need. No more, no less. You control the data model, the workflow, the integrations, and the roadmap. And critically: you decide what gets built next. Not a product manager in Sydney who’s never seen your workflows.

Running on cloud-native infrastructure like AWS means scalability, security, and availability are largely handled by the platform. The operational gap between “something you built” and “something a vendor hosts for you” has narrowed considerably. And the cost gap has flipped.

A custom-built equivalent to that €30,000/year SaaS tool, developed with AI-assisted tooling and running on serverless AWS infrastructure, might cost a fraction of that annually in operational expenses, after a one-time build investment that has also come down dramatically. More importantly: you own the asset. It does exactly what you need. And you’re not waiting seven years for someone to let you update an email address.

A Word of Honesty

Custom software isn’t free of responsibility. Someone has to maintain it, secure it, and evolve it. The SaaS argument, that someone else handles the infrastructure, the uptime and the patches, is not without merit. And for commodity functions like email, video conferencing, or payroll, that argument still wins. These are solved problems. Building your own would be an expensive act of reinvention.

But pairing custom-built applications with managed cloud services gives you the operational coverage of SaaS without the product dependency. You get uptime. You get security. You get to decide what comes next.

So What Should You Actually Do?

Not everything should be custom-built. That would be its own kind of madness.

The question worth asking is simpler: where are your core differentiating workflows? The processes that encode years of operational knowledge. The edge cases that keep showing up in feature request threads, yours and everyone else’s, year after year, status unchanged.

Those are exactly the features that will never quite make it onto a SaaS vendor’s roadmap.

The question is no longer “can we afford to build?” The question, in 2026, is whether you can afford to keep waiting.

JSDCLOUD-5746 would like a word.

The post Dear SaaS Vendor, We’ve Been Waiting Since 2017. appeared first on Cloudar.

Creating and managing secrets is perhaps as old as humans interacting with each other. Yet despite their critical importance, secrets remain one of the most vulnerable aspects of AWS infrastructure today. In our practice as Dev(Sec)Ops Engineers, we see these challenges daily accross our client environments.

To start, the question needs to be asked: “What exactly is a secret?”  In AWS and Cloud environments in general, a secret is anything used to access systems and/or data. These digital keys unlock potentially critical systems and must be protected from unauthorized access at all costs.

As Dev(Sec)Ops Engineers working in AWS, you are responsible for managing an extensive arry of secrets:

• RDS database usernames and passwords
• API keys for AWS services and third-party integrations
• CodeDeploy and deployment system credentials
• KMS encryption keys for data protection
• Private keys for SSL/TLS certificates and secure communications
• EC2 SSH keys and key pairs
• IAM user credentials for developers, QA, and operations teams
• Application service accounts and roles
• Any username/password combinations
• Sensitive configuration data that could aid attackers

The challenge is not just the volume … it’s the complexity of securing these secrets across multiple AWS accounts and other environments while maintaining operational efficiency.

The Hidden Dangers: How Secrets Get Compromised in AWS

 The Configuration File Trap

Storing secrets in configuration files is convenient but dangerous, especially in AWS environments where these files often end up in S3 buckets or EC2 instances. This risk was nicely illustrated in the RSA conference presentation “Red Team vs Blue Team on AWS” by Kolby Allen and Teri Radichel: Link to video

In the video Kolby deployed AWS resources using sample code found easily on the Internet.  Teri, simulating an attacker, conducted a penetration test and discovered a Lambda function’s configuration file stored in a S3 bucket (for convenience) complete with REDS database credentials in plain tekst.

If you must use configuration files in AWS:

• Maintain separate files for development, QA, and production AWS accounts
• NEVER commit these files to source control systems (GitHub, GitLab, Bitbucket)
• Use S3 bucket policies and encryption, but understand this is still suboptimal
• Consider AWS Systems Manager Parameter Store as a minimum improvement

This vulnerability is so important that it’s formally recognized in the MITRE attack framework:  [CWE-200: Exposure of Sensitive Information] (link)

Public Source Control: A Goldmine for AWS Account Takeovers

The most devastating secrets management failures occur when AWS credentials are committed to public repositories. The scale of this problem is staggering.

The people at TruffleHog constructed a live scanner for this:

https://forager.trufflesecurity.com/explore

This scanner continuously detects AWS access keys, secrets keys and other credentials in public Github commits, revealing the massive and constant stream of exposed AWS and other secrets.

This has a widespread impact: TruffleHog researchers discovered approximately 4,500 secrets among the top 1 million websites, many of which were AWS credentials, as detailed in their comprehensive analysis:

https://trufflesecurity.com/blog/4500-of-the-top-1-million-websites-leaked-source-code-secrets

Leaking these secrets can have far reaching financial consequences:

A Reddit user faced a 26.000 $ bill after IAM was compromised to execute crypto miners:

https://www.reddit.com/r/aws/comments/17p3v1e/account_got_hacked_and_get_26000k_bill/

A Developper was billed 14.000 $ on AWS following similar exposure:

https://dev.to/juanmanuelramallo/i-was-billed-for-14k-usd-on-amazon-web-services-17fn

AWS responded on these massive bills and is now actively scanning GitHub respositories through their AWS Credentials Exposed program and automatically disables discovery IAM access keys, but the frequency of these incidents remains alarmingly high as shown by the Trufflehog data.

Internal Systems: The False Security Blanket in AWS

Private repositories and internal systems create a dangerous illusion of security, even within AWS environments. The 2020 Twitter breach perfectly illustrates this vulnerability:

Attackers breached the perimeter, accessed internal Slack channels where developers had stored AWS credentials and other secrets, and used these to compromise infrastructure in a widely publicized incident:

https://www.zdnet.com/article/twitter-says-hackers-accessed-dms-for-36-users-in-last-weeks-hack

Secrets embedded in code even in fully private AWS services, proliferate across AWS services and do appear in:

• CloudWatch logs from Lambda functions and EC2 instances
• S3 bucket access logs
• CloudTrail event data
• Application Load Balancer logs
• Systems Manager Session Manager history

As such they do create additional attack vectors within your AWS environment.

Runtime Exposure: Secrets in AWS Production Workloads

Running AWS applications create numerous opportunities for secret exposure:

• Configuration files within EC2 instances or container images
• Environment variables visible in ECS task definitions or Lambda configurations
• Memory dumps from EC2 instances containing sensitive data
• Application caches in ElastiCache storing credentials
• CloudWatch logs revealing secrets in error messages
• EC2 instance metadata (IMDSv1) exposing IAM credentials
• Unencrypted S3 bucket metadata and tags
• AWS CloudShell command history
• Bash history on EC2 instances
• Container image layers in ECR with embedded secrets

This non-exhaustive list demonstrates how secrets can leak from multiple AWS vectors without proper handling.

What does AWS offer to fight this battle ? AWS Native Secrets Management

Adopt Just-in-Time Secret Retrieval with AWS Services

Core Principle: Store secrets in AWS-native management systems and retrieve them only when needed.

Instead of embedding secrets in configuration files, implement this AWS-secure workflow:

1. Store secrets in AWS Secrets Manager or Systems Manager Parameter Store
2. Retrieve secrets programmatically using AWS SDKs at runtime
3. Use IAM roles and policies for access control
4. Clear secrets from memory when no longer needed

AWS Access Control: Implement least-privilege IAM principles

• Developers cannot access production secrets via IAM policies
• Applications use IAM roles to retrieve only their required secrets
• Cross-account access is controlled via resource-based policies

AWS-Native Secrets Management Solutions

• AWS Systems Manager Parameter Store
• AWS Secrets Manager (Recommended for Production)

As an MSSP, we recommend AWS Secrets Manager as the gold standard for AWS environments.

Why Secrets Manager over Parameter Store?

• Enhanced security controls with fine-grained IAM integration
• Automatic secret rotation for RDS, DocumentDB, and Redshift
• Cross-account access capabilities essential for multi-account AWS strategies
• Comprehensive audit trails via CloudTrail integration
• No CloudFormation exposure risks unlike Parameter Store
• Encryption at rest using AWS KMS by default

As such we recommend the AWS Secrets Manager for Production workloads, the Encrypted Parameter Store can be used as a configuration store to fetch certain parameters.

How to use this in code ?

Python with boto3:

import boto3
import json

# Using IAM role-based authentication (recommended)
client = boto3.client('secretsmanager', region_name='us-east-1')

try:
    secret_response = client.get_secret_value(SecretId='prod/rds/mysql-credentials')
    secret_dict = json.loads(secret_response['SecretString'])
    
    # Use the secret
    db_username = secret_dict['username']
    db_password = secret_dict['password']
    
except ClientError as e:
    # Handle AWS-specific errors
    if e.response['Error']['Code'] == 'DecryptionFailureException':
        # Secrets Manager can't decrypt the protected secret text using the provided KMS key
        raise e
    elif e.response['Error']['Code'] == 'InternalServiceErrorException':
        # An error occurred on the server side
        raise e
    elif e.response['Error']['Code'] == 'InvalidParameterException':
        # Invalid parameter value
        raise e
    elif e.response['Error']['Code'] == 'InvalidRequestException':
        # Parameter value is not valid for the current state of the resource
        raise e
    elif e.response['Error']['Code'] == 'ResourceNotFoundException':
        # Can't find the resource that you asked for
        raise e

Via Infrastructure as Code:

• Cloudformation:
  • https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-secretsmanager-secret.html
• Terraform:
  • https://docs.aws.amazon.com/prescriptive-guidance/latest/secure-sensitive-data-secrets-manager-terraform/using-secrets-manager-and-terraform.html
• AWS CDK:
  • https://docs.aws.amazon.com/secretsmanager/latest/userguide/cdk.html
• Pulumi:
  • https://www.pulumi.com/registry/packages/aws/api-docs/secretsmanager/secret/

AWS Key Management Service (KMS)

AWS KMS provides dedicated encryption key management, solving the “where to store the encryption key” problem for AWS environments. KMS integrates seamlessly with Secrets Manager and most AWS services.

KMS Best Practices for Secrets:

• Use customer-managed KMS keys for production secrets
• Implement key rotation policies
• Use separate KMS keys per environment/account
• Leverage KMS key policies for fine-grained access control

AWS Systems Manager Parameter Store

While we recommend Secrets Manager for sensitive data, **Parameter Store** works well for:

• Non-sensitive configuration data
• Cost-conscious environments (free tier available)
• Simple use cases without rotation requirements

Multi-Account AWS Strategies

For AWS Organizations with multiple accounts (our recommended approach), consider:

Cross-Account Secrets Access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::PROD-ACCOUNT-ID:role/ApplicationRole"
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "secretsmanager:ResourceTag/Environment": "production"
        }
      }
    }
  ]
}

Multi-Region Considerations:

• Replicate critical secrets across AWS regions
• Use AWS Secrets Manager automatic replication
• Consider data residency requirements

AWS Implementation Best Practices from Our MSSP Experience

Security Architecture Considerations

• Design secure AWS deployment pipelines** using CodePipeline, CodeBuild, Github Actions with ci-cd plugins like trufflehog : https://undercodetesting.com/how-to-hunt-for-sensitive-credentials-using-trufflehog
• Implement comprehensive IAM access management with least-privilege principles
• Establish governance policies using AWS Config and AWS Organizations SCPs/RCPs
• Plan for secret rotation using AWS Secrets Manager automation
• Monitor and audit secret access via CloudTrail and CloudWatch

Operational Excellence in AWS

• Automate secret provisioning using AWS Lambda and CloudFormation/Terraform/…
• Implement emergency access procedures via AWS SSO and break-glass roles
• Establish incident response for compromised secrets using AWS Security Hub or other CSPM/CNAPP
• Regular security assessments using AWS Inspector and third-party tools
• Cost optimization by right-sizing Secrets Manager usage vs. Parameter Store

AWS-Specific Monitoring and Alerting

Set up CloudWatch alarms for:

• Unusual Secrets Manager API calls
• Failed secret retrievals
• Cross-account secret access
• KMS key usage anomalies

Conclusion

Effective AWS secrets management is not just about choosing the right AWS service it’s about implementing a comprehensive security strategy that leverages AWS-native capabilities while addressing the entire lifecycle of sensitive data. The examples and breaches discussed here represent real financial and reputational risks that AWS customers face daily.

Key AWS Takeaways:

1. Never store secrets in configuration files, S3 buckets, or source control
2. Use AWS Secrets Manager for production secrets management
3. Implement just-in-time secret retrieval with AWS SDKs
4. Apply least-privilege IAM policies and roles
5. Leverage AWS KMS for encryption key management
6. Plan for multi-account and multi-region secret strategies

The post AWS Secrets Management: Protecting Your Digital Keys in the Cloud appeared first on Cloudar.

Interestingly, Roles Anywhere does not completely solve the problem of not needing to create long-lived credentials, instead it moves (and reduces) the problem from credential management (rotation) to certificate management (distribution).

Setting up a Certificate Authority (CA) that can create certificates can be done with AWS Private Certificate Authority, and there are some options out there to manage distribution (usually as part of a bigger Device Management solution), but wouldn’t it be nice if the government of Belgium would do that for us?

The Belgian eID

In Belgium every citizens gets an identity card. The government makes sure that these get distributed and renewed, and since 2005 they contain a chip that holds (among other things) a certificate that can be used for digital authentication. If we can tie this to Roles Anywhere, we don’t have to worry about certificate management ourselves.

This eID is the size of a bankcard and the chip can be interacted with using off the shelf smart card readers. Most information can be read without any additional authentication – if you have the correct middleware, but to sign something you need to enter a PIN code.

The different certificates

Every eID holds an “authentication certificate” that is tied to the owner of the eID and has been signed by the “Citizens CA”.  The Citizens CA’s certificate is in turn signed by the Belgium Root CA . The Root certificate is self-signed. If we trust the Belgium Root CA, we will be able to validate any authentication certificate. More information about this setup can be found at https://repository.eidpki.belgium.be/#/home

The authentication certificate has two pieces of information that are going to be useful to us:

• The Issuer, showing us that the certificate is indeed issued by the Citizen CA.
• The Subject, containing the name and National Number (a number that uniquely points to one person) of the owner.

AWS IAM Roles Anywhere

Roles Anywhere allows you to use a certificate and its private key to get temporary credentials. You can think of it as an AssumeRole call, but instead of starting from an Access Key, we start from a certificate.

There are a few components we need:

• Configuration in our AWS Account, to indicate which CA we trust, and which roles can be assumed by different Issuer and/or Subjects. This also includes the creation of IAM Roles.
• Software on our computer to interact with the Card Reader so we can prove we have access to the eID and know its PIN.

For our use case we are going to trust the Belgium Root CA, and create a Role that can be assumed with any certificate issued by the Citizen CA. We will than limit the permissions of that IAM Role, so it can only write to a unique prefix in S3.

We can do that because every component of the Subject in our certificate gets mapped to a principal tag on the role session. E.g. if the subject is “C=BE, SN=Doe, GN=John, serialNumber=70010112345, CN=John Doe (Authentication)”, we have the following principal tags, and we can use principal tags in any policy.

• x509Subject/C: BE
• x509Subject/SN: Doe
• x509Subject/GN: John
• x509Subject/serialNumber: 70010112345
• x509Subject/CN: John Doe (Authentication)

Putting it all together

In our AWS Account

We will create the following resources in our AWS Account:

• An S3 Bucket
• A trust anchor, configured to trust the Belgium Root CA certificate that is publicly available via https://crt.eidpki.belgium.be/eid/brca6.crt.  This way we can use any certificate created by the Root CA of any of its subordinate CAs (like the Citizen CA)
• An IAM Role, where we configure the trust policy to only allow usage of the Role if:
  • The principal is the Roles Anywhere service (rolesanywhere.amazonaws.com)
  • The Issuer is Citizen CA (aws:PrincipalTag/x509Issuer/CN must equal “Citizen CA”)
  • The source is our trust anchor (aws:SourceArn equals the ARN of the Root CA trust anchor)
• An IAM Policy on this role that allows PutObject acces to all objects that start with the unique National Number (aws:PrincipalTag/x509Subject/serialNumber)
• a (Roles Anywhere) profile that ties the trust anchor to the Role. We could configure more settings here, like a mapping of fields from the certificate or a session policy, but don’t need to.

Of course we do not have to do this by hand, we create a CloudFormation template for this. It can be found in our sample repository: https://github.com/WeAreCloudar/cloudformation-samples/tree/main/templates/roles-anywhere-eid

On our machine

On our machine, we will need – besides the eID and its PIN (at least on macOS, where I tested this):

• a SmartCard Reader
• the Roles Anywhere Credential Helper (downloadable from the AWS Website)
• The AWS CLI (or any other tool we want to use with our AWS Credentials)

We also need a few pieces of information:

• The ARNs of the trust anchor, profile and role we created
• Our own National Number and the name of the bucket (to test our policy)

Ideally we’d be able to configure the credential helper to read any eID we insert, and this is possible if you install the Belgian eID Middleware using the following command.

aws_signing_helper credential-process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--certificate "pkcs11:model=Belgium%20eID;object=Authentication" \
--pkcs11-lib '/Library/Belgium Identity Card/Pkcs11/beid-pkcs11.bundle/Contents/MacOS/libbeidpkcs11.dylib

However because of an issue with the helper, this only works on macOS if you build the helper yourself from source. Since we do not want to do that, we can plug in our eID and find and identifier using “aws_signing_helper read-certificate-data”. You should get something like

Matching identities
1) bf013dd57aa34f8f9d4e22ab89dd8eb2 "SERIALNUMBER=70010112345,CN=John Doe (Authentication),C=BE,..."
2) 786f7572198a4a8a82ee7ef3f205d7be "SERIALNUMBER=70010112345,CN=John Doe (Signature),C=BE,..."


Copying everything between quotes, allows us to create a new file, selector.json that looks like this:


[
{
"Key": "x509Subject",
"Value": "SERIALNUMBER=70010112345,CN=John Doe (Authentication),C=BE,..."
}
]

and use that to select our specific certificate:

aws_signing_helper credential-process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--cert-selector file://selector.json

Our smart card reader will now prompt us for a PIN. If we enter the right one, we get credentials as output:

{"Version":1,"AccessKeyId":"ASIA...","SecretAccessKey":"COA...","SessionToken":"IQoJ...","Expiration":"2025-03-28T14:16:07Z"}


We can put this command in our ~/.aws/config file, so the AWS CLI will execute it for us, but for testing purposes it’s slightly easier to use the “serve” option.  The AWS Documentation has a longer explanation about how you can use the credential-process: https://docs.aws.amazon.com/rolesanywhere/latest/userguide/credential-helper.html#credential-helper-examples

Testing with s3

In one terminal we can run a credential / metadata server that can be used by the AWS CLI:

~$ aws_signing_helper serve -process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--cert-selector file://selector.json


This gives us
2025/03/28 14:28:12 Local server started on port: 9911
2025/03/28 14:28:12 Make it available to the sdk by running:
2025/03/28 14:28:12 export AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:9911/

We can than use a second terminal to run:

AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:9911/ aws s3 cp example_file s3://$bucket/serialNumber/70010112345/example_file

and get our file in s3:

upload: example_file to s3://.../serialNumber/70010112345/example_file

Conclusion

This experiment shows that being able to use an external device withIAM Roles, is a very powerful feature, because it can remove a lot of tasks related to certificate and credential management. Being able to do this with something I had in my backpocket made it extra cool.

However, the Belgian eID is probably not the best way to go about this in a real scenario:

• I just gave everyone living in Belgium access to my S3 Bucket. I could use a more scoped-down trust policy, but I would eventually run into limitations of trust-policy size.
• I also didn’t tackle Certificate Revocation Lists (CRL). Every time someone loses their wallet, I would need to invalidate certain certificates – this also runs into size limitations
• I already have a good way to give users access to AWS Accounts, using Identity Center or Cognito.
• The pkcs11 support on macOS seems limited, and I would have to install the middleware everywhere,

This does not mean that Roles Anywhere is not a good solution for this kind of cases:

• If you are already putting certificates on machines (e.g using an MDM solution), AWS IAM Roles Anywhere is a great fit
• Instead of relying on the Belgian Government you can buy an hardware key like a yubikey and use it as your certificate source. This is as secure as using the eID (only the machine with the key plugged in will be able to request credentials), and you would be limiting it to the exact serial number(s) in your trust policy anyway – making CRLs mostly moot.

The post Sign in with your eID: Using AWS IAM Roles Anywhere with a SmartCard Reader appeared first on Cloudar.

Cloudar has achieved the AWS Global Security & Compliance Acceleration (GSCA) certification, underscoring our commitment to providing secure, compliant cloud solutions. For AWS users, this milestone brings several benefits, helping them streamline operations, meet compliance requirements, and ensure the highest levels of security.

About the AWS Global Security & Compliance Acceleration Program

AWS designed the GSCA program to support partners in strengthening the security and compliance of cloud environments. Through the GSCA program, we have gained access to advanced training, resources, and AWS experts, enabling us to fine-tune our security capabilities and stay ahead of ever-evolving threats.

This program emphasizes automation, governance, and continuous monitoring, empowering us to proactively address security challenges and scale securely in the cloud.

Enhanced security posture

With this certification, Cloudar can provide customers with ready-to-go security solutions, customized to fit their unique requirements. From secure infrastructure setup to automated incident response, we can ensure your workloads are protected from day one.

By adhering to AWS’s best practices, we can ensure that your infrastructure is secure by design. We leverage tools like AWS Security Hub, Amazon GuardDuty and AWS Config to provide customers with continuous threat detection, rapid incident response, and real-time insights into security events. This means your business can identify and mitigate risks before they escalate.

Regulatory compliance

Compliance is no longer an afterthought. Cloudar uses AWS-native tools to continuously monitor for regulatory compliance and offers pre-built templates to help meet industry standards, reducing the time and effort required to maintain compliance.

Whether your business operates in a heavily regulated industry like healthcare, finance, or government, the GSCA certification ensures that we can help you meet key regulatory requirements such as GDPR, HIPAA, SOC 2, and more.

Faster time-to-market:

By automating security and compliance processes such as patch management, configuration drift detection, and log monitoring, we can significantly reduce the time it takes to implement secure, compliant workloads on AWS.

Cloudar ensures that our customers can maintain a high security posture without the overhead of manual intervention. This leads to lower operational costs and more efficient resource allocation, so you can focus on innovation and growth, while we handle the complexities of cloud security.

Cloudar helps you leverage the power of AWS Security

Cloudar’s approach goes beyond simply implementing AWS’s built-in tools. We work closely with your teams to tailor solutions that fit your specific business needs, ensuring that you get the most out of the AWS cloud while keeping your data secure and compliant.

We design cloud architectures that follow AWS’s security best practices, but are customized to fit the scale, complexity, and compliance needs of your business.

Whether you’re migrating to the cloud or optimizing existing workloads, Cloudar ensures a smooth integration of AWS security services with your current operations.

But our support doesn’t stop after deployment: Cloudar continuously monitors your cloud environment, ensuring you stay compliant and secure as your business grows.

The Cloudar Difference

At Cloudar, we understand that security and compliance aren’t just checkboxes – they’re critical to building trust with your customers and staying ahead in a competitive market.

By achieving the Global Security & Compliance Acceleration on AWS, Cloudar once again proves that we are not just a partner, but a trusted ally in your cloud journey.

We provide peace of mind that your AWS workloads are not only running efficiently but are also secured against threats and aligned with the latest regulatory standards.

If you’d like to learn more about how Cloudar can enhance your cloud security and compliance, or if you’re ready to take your AWS workloads to the next level, don’t hesitate to get in touch with our team today!

The post Achievement Unlocked: Global Security & Compliance Acceleration on AWS appeared first on Cloudar.

At Cloudar, we’ve always believed that security should be at the heart of every cloud strategy. Our commitment to delivering top-notch cloud security solutions has led us to achieve numerous milestones, and today, we’re excited to announce our latest achievement: Cloudar has earned the prestigious AWS Security Competency!

Building on our ISO/IEC 27001 certification since 2018 and our existing AWS competencies, this accomplishment marks a significant milestone in our journey to provide the highest level of security and service to our customers. In this blog post, we’ll dive into what the AWS Security Competency means for us and, more importantly, what benefits it brings to our valued customers.

What is AWS Security Competency?

The AWS Security Competency is not just a fancy title; it’s a recognition of excellence in designing, implementing, and managing security solutions on the Amazon Web Services platform. AWS awards this competency to organizations that demonstrate exceptional proficiency in securing AWS workloads and infrastructure.

Why it matters

Expertise You Can Trust

Achieving the AWS Security Competency is a rigorous process that involves a deep evaluation of an organization’s expertise in various security domains such as identity and access management, data protection, compliance, and more.
This achievement is a seal of approval from Amazon Web Services itself.
It means that Cloudar’s team of experts possesses a comprehensive understanding of AWS security best practices, threat detection, and incident response. It signifies that we have demonstrated proficiency and skill in protecting your cloud infrastructure and data, giving you peace of mind that your assets are in capable hands.

Cutting-Edge Solutions

Our team at Cloudar stays at the forefront of security technology. Achieving the AWS Security Competency ensures that we are equipped with the latest knowledge and tools to keep your AWS environments secure from evolving threats.
Our partnership with Trend Micro played a crucial role in this accomplishment. Trend Micro is a global leader in cloud security, and our collaboration has allowed us to leverage their cutting-edge technology and expertise to enhance our security solutions. Together, we’ve created a formidable team dedicated to safeguarding our customers’ cloud environments.

Dedication to Continuous Improvement

Security is not a static field; it’s an ever-evolving landscape. To maintain this competency, Cloudar continues to invest in training and staying up-to-date with the latest security trends and technologies. We are committed to providing our customers with the most advanced security solutions available.

Proven Track Record

Our attainment of this competency underscores our extensive experience in designing, implementing, and managing secure AWS solutions.
We understand that one-size-fits-all security solutions rarely work. Cloudar’s expertise allows us to tailor security measures to your unique needs, ensuring that your AWS environment is both secure and efficient. We’ve successfully tackled complex security challenges for customers across various industries, consistently delivering exceptional results.

Benefits for our customers

Compliance Assurance

As an ISO/IEC 27001 certified company, Cloudar understands the importance of compliance.
With Cloudar’s AWS Security Competency, you can trust us to help you meet and exceed regulatory compliance requirements. Our deep understanding of AWS services ensures that your cloud infrastructure aligns with industry-specific standards, making audits and certifications a breeze.

Enhanced Security Posture

With our AWS Security Competency and other AWS competencies, you gain access to top-tier security practices and expertise.
We go above and beyond to ensure your AWS environments are fortified against potential threats and vulnerabilities, significantly reducing the risk of data breaches and disruptions.

Cost Savings

Investing in security is an investment in your business’s longevity. By preventing costly breaches and downtime, Cloudar’s expertise can save you substantial financial resources in the long run.

Increased Efficiency

Our tailored security solutions are designed to integrate seamlessly with your existing AWS environment. This ensures that security measures do not impede your operations, allowing for increased efficiency and productivity.

Peace of Mind

In the event of a security incident, Cloudar’s well-practiced incident response teams are ready to spring into action. Our proactive security measures help mitigate threats before they become major issues, minimizing downtime and data loss.
With Cloudar as your trusted AWS Security Competency partner, you can focus on growing your business, knowing that your cloud infrastructure is fortified against threats. Sleep better at night, knowing that experts are on guard.

Conclusion

Achieving the AWS Security Competency is not just an accolade for Cloudar; it’s a testament to our unwavering dedication to providing the highest level of security for our customers’ AWS environments.

We are committed to keeping your data safe, your operations efficient, and your business thriving. Your cloud’s security is our priority, and together, we’ll reach new horizons.

To learn more about how Cloudar can secure your AWS environment and help your business grow securely, contact us today and experience the difference of working with an AWS Security Competency holder!

The post Elevating cloud security to new heights: Cloudar earns AWS Security Competency! appeared first on Cloudar.

As the Internet evolved from a small interconnect between universities where every resource had a public IP address, and resources were managed based on mutual respect and consensus to the Internet as we know it today with multiple threads, risks, and shortage of IP addresses, the direct and public accessibility of all resources could no longer be sustained.

As stated in the security pillar of the Well Architected Framework, the main security principle in AWS Cloud is to create several network layers and defense in depth.

The framework states more specifically: “Do not place resources in public subnets of your VPC unless they absolutely must receive inbound network traffic from public sources” SEC05-BP01 Create network layers.

So as a best practice, cloud engineers strive to deploy their resources in private subnets whenever possible. Because of the confidential nature of these subnets, engineers had to invent ways to manage their resources in these private subnets.

2. How to Use Jump / Bastion Hosts to Manage Your Private Subnets

A solution to manage the infrastructure in your private subnets is to deploy a bastion host. Before 2018, this was the recommended way, and it was well-documented:

For Linux Workloads:
https://aws.amazon.com/solutions/implementations/linux-bastion/

For Windows Workloads:
https://aws.amazon.com/solutions/implementations/rd-gateway/

Although these solutions do solve the problem stated above (how to manage your Linux and/or Windows workloads in their private subnets), these solutions have several drawbacks:

• These solutions still need a public IP address to be associated with the virtual machines and open ports to enable remote access.
• The Linux solution is vulnerable to SSH multiplexing access.
• As with all software that is publicly accessible, they need to be strictly maintained and patched:
  • https://www.bleepingcomputer.com/news/security/remote-desktop-zero-day-bug-allows-attackers-to-hijack-sessions/
  • https://www.microsoft.com/en-us/security/blog/2021/09/02/a-deep-dive-into-the-solarwinds-serv-u-ssh-vulnerability/
• It’s hard to capture the remote access’s metadata and/or full session logs.
• Password management and access control are needed for the management users.

Because IT is all about answering questions, we are looking for an answer to this question: ‘How can I manage my resources in private subnets in a secure, encrypted, and auditable fashion?’ Luckily, AWS came up with two solutions that answer this question.

3. AWS Systems Manager’s New Session Manager

AWS listened to the drawbacks inherent to using bastion/jump hosts and introduced the Session Manager service as a component of AWS Systems Manager. The Session Manager supports both inside AWS and on-premises:

• Windows
• Linux
• MacOS

This blog post will focus on using it with Elastic Compute Cloud (EC2) Linux instances running in AWS.

3.1 Why AWS Session Manager

Session Manager solves the problem mentioned above: to access instances within private subnets that don’t allow ingress connections from the Internet. This is made possible by the Systems Manager (SSM) agent running on the EC2 instances, which pushes traffic to the Session Manager service managed by AWS outside your EC2 environment. The configuration shown in the diagram below provides an example of how it can be configured:

As you can see in the diagram above, the client machine is outside of AWS. It accesses the internal machines over the Internet using valid user credentials for the AWS environment. To access the internal machines, the client can use several methods:

Via the AWS console, you have two-fold access:

• The AWS Systems Manager console includes access to all the Session Manager capabilities for both administrators and end users.

• The Amazon EC2 console allows end users to connect to the EC2 instances for which they have been granted session permissions.

• Via the AWS Command Line Interface (CLI): this requires the installation of the Session Manager Plugin. It is possible to restrict how users launch sessions, so you could choose only to allow them to invoke a connection via the AWS CLI if your users don’t have access to the console or the other way around.

• The Session Manager SDK consists of libraries and sample code that allows application developers to build front-end applications, such as custom shells or self-service portals for internal users that natively use Session Manager to connect to managed nodes. This allows you to integrate Session Manager into your own tooling or automation workflows.

3.2 Configuration

3.2.1 Network Considerations

As shown in the diagram above, the SSM agent actually uses HTTPS to communicate with the SSM endpoint. As such, port 22 for SSH or port 3389 for RDP are no longer required and should not be allowed as ingress traffic. To make this even more secure, we can use VPC endpoints: they use AWS PrivateLink, so traffic between the target EC2 instances and the Session Manager service does not traverse the Internet. To enable this, create these VPC endpoints in your target VPCs, replacing the region accordingly, for example, eu-west-1

Interface endpoints:

• com.amazonaws.[region].ssm

• com.amazonaws.[region].ssmmessages

• com.amazonaws.[region].ec2messages

• com.amazonaws.[region].ec2 [For Windows workloads]

Optional but recommended:

• com.amazonaws.[region].kms = To encrypt logs in CloudWatch

• com.amazonaws.[region].logs = To send logs to CloudWatch

Gateway endpoint:

• com.amazonaws.region.s3

Systems Manager uses this endpoint to update SSM Agent and to perform patching operations. Systems Manager also uses this endpoint for tasks like uploading output logs that you choose to store in S3 buckets, retrieving scripts or other files you store in buckets, and so on.

If you restrict egress networking on the EC2 machines via security groups, you should use the prefix list of the S3 gateway endpoint to send traffic to this gateway endpoint only. Here is an example of security group configuration for a tightly controlled EC2 instance:

As the interface endpoints replace the DNS addresses of the SSM services, the network traffic with the configuration listed above only flows over AWS PrivateLink. You can even further limit access to the S3 buckets that SSM uses by using the structure listed in the AWS documentation: S3 bucket list

3.2.2 EC2 Prerequisites and IAM Configuration

EC2 Prerequisites

The EC2 instance should run the Systems Manager (SSM) agent to support the Systems Manager. The Amazon Linux AMIs support the agent by default, and it can be installed manually on other Linux systems by installing the Systems Manager. The agent allows a lot more than just connecting with Session Manager. More information about its capabilities is available here. Even if you want to use it on a system that is not currently supported, you can always download the open-source code and modify it yourself. However, AWS will not support a modified version of the agent.

IAM Configuration

Each EC2 Instance must have permission to make API calls to the Session Manager Service. All necessary permissions are available in the Amazon-managed IAM policy: AmazonSSMManagedInstanceCore, and this policy should be attached to an Instance profile.

The policy listed above gives you a good baseline, but if you want to lock this down further, you can look up the information here.

Using this method to access the instances does not require any management of SSH keys. Access is only based on IAM policies since everything is managed through the Session Manager.

When configuring Session Manager access for your end-users, you can limit which instances they can access, whether it’s through the command line or the console, and if they are allowed to tunnel SSH or just use the regular Session Manager shell access.
More information about how to configure the IAM policies is available here.

Some AWS policy templates show how to restrict access to specific EC2 instances, but that will probably not be scalable if you really want to implement this for your organization.

Instead, you can refer to the templates available here to see how to configure the policies to allow access based on tags. As shown here, you will also want to ensure you add permissions for users to terminate only the sessions they started. Otherwise, a user may be able to terminate another user’s session.

3.3. Logging

3.3.1. What the AWS CloudTrail Does

Without configuring anything, AWS CloudTrail will collect basic information about sessions. When someone launches a remote access session with Session Manager, SSM will log an event named “StartSession.” This event will include a number of interesting things, such as:

• The username that launched the session
• Whether the user was authenticated with multi-factor authentication (MFA)
• The originating IP address
• The unique ID of the target EC2 instance
• The session ID

The events in CloudTrail are helpful for getting a sense of who’s logging into your instances and when it’s happening. However, it does not provide any information about what they are doing once they’ve established a session.

3.3.2. Session logs

As mentioned in the overview of AWS Session Manager, SSM uses HTTPS to establish sessions, but it is also possible to use it as a tunnel for other protocols, such as SSH or RDP. Be aware that AWS cannot provide full session logs for those connections if you use the tunneling option. The diagram below shows an example configuration for logging full session data either to CloudWatch or S3 when tunneling is not used:

The logs in CloudWatch look like this:

Next to the CloudWatch logs, logging into S3 is also possible.

Please make sure that you attach a policy to your instance profile that allows writing to the S3 bucket you configured to store your session data. The result is that you will see log files written to the specified bucket with the name of the session (the username and a unique string). The logs contain the commands typed, along with their output, which is the same as what is shown in CloudWatch.

3.4. Conclusion

The question: “How can I manage my resources in private subnets in a secure, encrypted, and auditable fashion?” is answered by using the Systems Manager Sessions Manager.

4. Instance Connect and Instance Connect Endpoint

As there are often multiple ways to reach the same goal, AWS delivers a different option to answer the same question: “How can I manage my resources in private subnets in a secure, encrypted, and auditable fashion.” This can also be solved by using Instance Connect with its newest addition: Instance Connect Endpoint

4.1. What Is Instance Connect?

The Instance Connect feature was introduced in 2019 and offers a solution to control SSH access to your instances using AWS Identity and Access Management (IAM) policies and audit connection requests with AWS CloudTrail events. In addition, you can leverage your existing SSH keys or further enhance your security posture by generating one-time use SSH keys each time an authorized user connects.

Instance Connect works with any SSH client, but you can also easily connect to your instances from the EC2 Console.

The question: “How can I manage my resources in private subnets in a secure, encrypted, and auditable fashion?” is not solved by this approach because:

• The machine still needs a public IP address.
• The connection logs cannot be stored in S3 or CloudWatch.

A few months ago, AWS introduced a new Instance Connect Endpoint feature that solves the first drawback listed above.

4.2. Instance Connect Endpoint

With the Endpoint for Instance Connect (EIC) Service, you no longer need a public IP address on your resource or any agent to connect to your resources. The Instance Connect Endpoint Service works in three ways:

• AWS Management Console
• AWS Command Line Interface (AWS CLI)
• SSH Clients like PuTTY or OpenSSH

4.2.1. Instance Connect Endpoint Features

The EIC Endpoint Service is an identity-aware TCP proxy. As described above, it has two modes:

• AWS CLI | SSH CLients
• AWS Console

The AWS CLI client is used to create a secure WebSocket tunnel from your workstation to the endpoint, where authorization happens with your AWS Identity and Access Management (IAM) credentials. When the tunnel is established, point your preferred SSH client at your loopback address (127.0.0.1 or localhost) and connect as usual.

Second, the Console gives you secure and seamless access to resources inside your VPC. Authentication and authorization are evaluated via IAM before traffic reaches the VPC. The diagram below illustrates a user connecting to an AWS EIC.

As shown above, the EIC Endpoints Service provides a high degree of flexibility.

• It doesn’t require your VPC to have direct Internet connectivity
• No agent is needed on the resource(s) you wish to connect to
• It preserves existing workflows by allowing your preferred client software to connect
• IAM and Security Groups can be used to control access

4.2.2. Instance Connect Endpoint Security and Logging

IAM

The beauty of this solution lies in the fact that access can be limited to what the user needs: EIC Endpoints Service follows the important security requirements in terms of the separation of privileges for the control plane and the data plane. An administrator with full EC2 IAM privileges can create and control EIC Endpoints (the control plane). However, the same administrator cannot use those endpoints without also having EC2 Instance Connect IAM privileges (the data plane). The DevOps Engineers who may need to use EIC Endpoint to tunnel into VPC resources do not require control-plane privileges to do so.

CloudTrail

Records of data plane connections include the IAM principal making the request, their source IP address, the requested destination IP address, and the destination port. An example of connecting to the instance from the console:

{
“eventVersion”: “1.08”,
“userIdentity”: {
“type”: “FederatedUser”,
“principalId”: “123456789:Example”,
“arn”: “arn:aws:sts::123456789:ExampleUser”,
“accountId”: “123456789”,
“accessKeyId”: “ExampleKey”,
“sessionContext”: {
“sessionIssuer”: {
“type”: “IAMExample”,
“principalId”: “ExampleID”,
“arn”: “arn:aws:iam::123456789:user/exampleuser”,
“accountId”: “123456789”,
“userName”: “exampleuser”
},
“webIdFederationData”: {},
“attributes”: {
“creationDate”: “2023-07-24T12:24:55Z”,
“mfaAuthenticated”: “true”
}
}
},
“eventTime”: “2023-07-24T13:19:22Z”,
“eventSource”: “ec2-instance-connect.amazonaws.com”,
“eventName”: “SendSSHPublicKey”,
“awsRegion”: “eu-west-1”,
“sourceIPAddress”: “123456789”,
“userAgent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0”,
“requestParameters”: {
“instanceId”: “i-123456789”,
“instanceOSUser”: “ec2-user”,
“sSHPublicKey”: “ssh-ed25519Example\n”
},
“responseElements”: {
“requestId”: “b2a9eeae-7189-4b6c-ac8a-a41e56ea1a42”,
“success”: true
},
“requestID”: “b2a9eeae-7189-4b6c-ac8a-a41e56ea1a42”,
“eventID”: “161427f7-8f1e-459a-83f1-2f54aca879ca”,
“readOnly”: false,
“eventType”: “AwsApiCall”,
“managementEvent”: true,
“recipientAccountId”: “123456789”,
“eventCategory”: “Management”,
“tlsDetails”: {
“tlsVersion”: “TLSv1.3”,
“cipherSuite”: “TLS_AES_128_GCM_SHA256”,
“clientProvidedHostHeader”: “ec2-instance-connect.eu-west-1.amazonaws.com”
},
“sessionCredentialFromConsole”: “true”
}

4.3. Connection Examples

4.3.1. Via the Console

This gives you:

As you can see, the console does not login via SSM-user but via EC2-user and with a private IP.

4.3.2 Via the AWS CLI

To connect via the AWS CLI, make sure that you have the latest AWS CLI installed. Then you can use the new ec2-instance-connect ssh command from the AWS CLI. With this new command, AWS generates ephemeral keys for you to connect to your Instance.

Note that this command requires using the OpenSSH client and the latest version of the AWS CLI. You need IAM permissions, as detailed here, to use this command and connect. You can find an example below:

4.3.3 Via SSH Clients

Create your own private key:

ssh-keygen -t rsa -f mynew_key

Use the following AWS CLI command to authorize the user and push the public key to the Instance using the send-ssh-public-key command. To support this, you need the latest version of the AWS CLI.

aws ec2-instance-connect send-ssh-public-key –region eu-west-1 –instance-id i-0123456789example –availability-zone eu-west-1a –instance-os-user ec2-user –ssh-public-key file://mynew_key.pub

This gives back:

{
“RequestId”: “9c4bf3df-799d-4f40-9e2d-cbc3ed3bbe08”,
“Success”: true
}

After authentication, the public key is made available to the Instance through the Instance metadata for 60 seconds. During this time, connect to the Instance using the associated private key:

ssh ec2-user@[INSTANCE] \
-i [SSH-KEY] \
-o ProxyCommand=’aws ec2-instance-connect open-tunnel \
–instance-id %h’

This gives back:

With this method, plain SSH clients can still be used via short-lived keys.

4.3.4 Connecting to the Windows Machines

How to connect to Windows machines is similar:

aws ec2-instance-connect open-tunnel \
–instance-id i-0123456789example \
–remote-port 3389 \
–local-port any-port

In your RDP Client, set it up like this, mark that the –local-port is set to 5555

4.4 To Sum Up Instance Connect

As shown in the blog post above, Instance Connect allows you to connect to your private resources privately and securely without needing long-lived SSH keys and/or bastion hosts.
The drawback compared to AWS Sessions Manager is that you cannot log session logs towards Amazon S3 or Amazon CloudWatch. The benefit is that you don’t need to install custom agents on your machines, which makes this solution the preferred solution for older Instances and appliances running in your AWS environment.

5. Final Conclusion

AWS provides various methods to connect to Instances, including SSH, RDP, AWS Systems Manager Session Manager, EC2 Instance Connect and Endpoint, bastion hosts, and VPN. The choice of method depends on the operating system, security requirements, network configuration, and personal preference. By leveraging these connection methods, you can securely access and manage your AWS Instances based on your specific use cases and requirements.

What questions do you have after reading this blog post? We would love to answer them!

The post Managing Instances in Private Subnets with Session Manager or Instance Connect appeared first on Cloudar.

#1 Develop a comprehensive strategy

Before deploying your applications on AWS, it is essential to establish a well-defined cybersecurity strategy. This strategy should include conducting a risk assessment, performing threat modelling, and devising a mitigation plan. Identify potential risks and vulnerabilities specific to your cloud infrastructure and implement measures to address them. This will ensure compliance with industry standards and regulations while proactively preventing cyberattacks.

#2 Become familiar with the AWS Well-Architected Framework

AWS offers abundant resources to help organizations enhance their security posture. The AWS Well-Architected Framework provides guidance for designing and operating secure and cost-effective systems in the cloud. The Framework encompasses five key pillars: operational excellence, security, reliability, performance efficiency, and cost optimization. Understanding and leveraging the Framework is key to safeguarding applications against potential cyberthreats.

#3 Implement robust security controls

Taking proactive measures to impose strong cloud security controls is crucial for organizations to secure their AWS cloud infrastructure effectively. As the responsibility falls on you to protect your cloud workloads, consider implementing security controls to minimize the risk of security breaches, including clearly defining user roles, conducting privilege audits, enforcing a strong password policy, and using multi-factor authentication (MFA). Consistent enforcement and adherence to these security controls throughout your organization are essential.

#4 Ensure easy access to security policies

Accessibility to your AWS security policies is key to promoting a culture of security within your organization. These policies should encompass access control, network security, encryption, and incident response guidelines. Making policies accessible helps everyone in your organization understand the importance of security and act in accordance with your policies.

#5 Safeguard data with encryption

Encrypting your data prevents unauthorized access and ensures its confidentiality, even if intercepted or stolen. Encryption is often mandatory for regulatory compliance. AWS provides various encryption options, including server-side encryption, client-side encryption, and transit encryption. Familiarize yourself with these options and choose the appropriate method to secure your sensitive data within your cloud environment.

#6 Back up data consistently

Regular data backups ensure data integrity and availability. In the event of cyberattacks, hardware failures, or accidental deletions, having up-to-date backups provides peace of mind and enables data recovery. The frequency of backups should be based on the criticality and volatility of your data. Regulatory requirements and business needs will determine how long the data will be retained. AWS offers various backup options, such as Amazon EBS, Amazon S3, and AWS Backup. By leveraging these options, you can store your backups in multiple regions to ensure constant protection.

#7 Stay up to date

AWS regularly releases security patches, bug fixes, and updates to address vulnerabilities and counter new threats. Keeping your AWS systems up to date maintains the security of your environment and protects against potential attacks. Take advantage of AWS’s automatic update features, such as Amazon Inspector and AWS Systems Manager, to automate patching and ensure your systems are always current. Configure notifications to receive alerts about new updates, enabling prompt action to stay ahead of security risks.

Want to optimally prepare your organization for cloud-related security threats? Get in touch with Cloudar, an official Next Generation AWS Managed Service Provider Partner.

The post Get it covered: 7 essential practices for AWS security appeared first on Cloudar.

The findings

When you configure an AWS service to use an IAM role on your behalf, you need to have the iam:PassRole permission in addition to the permissions required to configure and/or use the service.

Due to the bug we reported, AWS Directory Service didn’t verify the presence of those permissions when assigning users or groups to an existing IAM role. This allowed users with the  ds:EnableRoleAccess permission to assign users to any role in their account that has a trust relationship with AWS Directory service.

Additionally, we found some actions that were not logged to CloudTrail and a validation that’s only done client-side

The discovery

Cloudar is an AWS Managed Service Provider Partner, and as such, we work together with our clients to configure their environments. One of our clients alerted us of a strange behavior in the AWS Console for AWS Directory Service, which prompted us to take a deeper look behind the scenes of how that service works.

Step 1: Missing CloudTrail Data

We had configured AWS Directory Service for AWS Management Console access for our client, and it was working properly for them. However, when looking in the console, the feature appeared to be disabled.

Usually, we would investigate CloudTrail logs to see what disabled this, but at the time, we didn’t see an entry in CloudTrail that showed the enabling or disabling of access. When we tested what happened when changing the status ourselves, we also did not see those actions recorded in CloudTrail.

We’ve reported this to AWS, and enabling or disabling console access does now create events in CloudTrail (note: **OMITTED** is part of the CloudTrail event):

{
[…]
    "eventSource": "ds.amazonaws.com",
    "requestParameters": {
        "directoryId": "d-123456789a",
        "applicationId": "***OMITTED***",
        "serviceAccountUserName": "***OMITTED***",
        "serviceAccountPassword": "HIDDEN_DUE_TO_SECURITY_REASONS"
    },
    "responseElements": null,
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "sessionCredentialFromConsole": "true"
}

{
[…]
    "eventSource": "ds.amazonaws.com",
    "eventName": "UnauthorizeApplication",
    "requestParameters": {
        "directoryId": "d-123456789a",
        "applicationId": "***OMITTED***"
    },
    "responseElements": null,
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "sessionCredentialFromConsole": "true"
}

Step 2: Looking at what the console does

Instead of looking through CloudTrail logs, we opened the development console in our browser. There we saw that it was sending requests to an undocumented API when enabling console access. There is no public API for enabling console access or assigning user to roles within Directory Service, so this wasn’t unexpected.

What was unexpected was that when enabling or disabling console access, there were two calls to the internal API. One to (un)authorize an application called “AWSManagementConsole” and one to (un)authorize an application called “Directory Service console”.

Assuming there was a consistency problem, we investigated what happened if we only unauthorized one of them by sending requests to the undocumented API from the command line.  When we unauthorized the “AWSManagementConsole” and authorized “Directory Service console”, we were able to reproduce our customer’s issue where the console would show “disabled”, but the sign-in would still work.

By the time we sent our report to AWS, the double API call didn’t happen anymore (only “AWSManagementConsole” is used today), so this seemed to have been a temporary issue. We still reported this behavior, indicating that we could no longer reproduce.

Step 3: Finding a race condition

Since this wasn’t an easy bug to uncover, we had to read a lot of documentation to deeply understand what Directory Service was doing. While doing that, we noted an interesting sentence: “If any IAM roles have been assigned to users or groups in the directory, the Disable button may be unavailable.”

The word “may” seemed very ambiguous, especially after finding an issue where a network disconnection could lead to an unexpected state. We looked a bit further and realized that the expected behavior is that you can’t disable the integration if you still have users and/or groups assigned to roles.

However, that check is done completely in the browser, and appears to be based on the state when the tab was loaded. This means you can disable the console access while still having users and/or groups assigned. You could do this by calling the same (undocumented) API that the console uses, or in the browser by 1) opening the console while there are no assignments 2) using a second tab to assign users and/or groups to a role 3) using the first tab to disable console access.

We’ve reported this issue to AWS, but don’t consider it a vulnerability as the assignments are not usable if the Console access is disabled.

Step 4: Privilege escalation

Having discovered two APIs that did some part of their logic client-side, and having experimented with the role-assignment, we looked at other validations that should happen when you assign a Role to a user in a directory.  To assign a specific role to a user or group:

• you should have ds:EnableRoleAccess permission for the Directory Service domain;
• the role you want to pass should have a trust policy that allows use in AWS Directory Service;
• the role should be in the same AWS account;
• you should have iam:PassRole permissions on the role.

When we tested these checks, we noticed that we could assign a role (within the same account), even if we didn’t have the right iam:PassRole permission. In certain setups, this could mean privilege escalation.

• Imagine you have two people, Alice and Bob, managing a domain, and each of them is responsible for assigning users to specific IAM roles. Alice assigns people to Role A and Bob assigns people to Role B. They could abuse the missing iam:PassRole check to assign people to a role they are not responsible for. For example, Alice assigns herself (or someone else) to Role B.
• Another possible scenario would be when a user has the required permissions to create and manage a domain in AWS Directory Service, but no IAM permissions (e.g., with the PowerUserAccess managed policy). If there is already a role that has a trust relationship with AWS Directory Service, that user could create their own domain and assume that existing role.

This issue has been completely fixed.

Recommended actions

Taking advantage of this vulnerability required a specific configuration and a high level of permissions within Directory Service. AWS customers who were relying on iam:PassRole permissions for access control before 2023-05-12 should review their directory for misconfiguration and their CloudTrail logs for unexpected AssumeRole calls.

Timeline

2023-04-07: Issue found and reported to AWS.

2023-04-07: Investigation started by AWS.

2023-04-13: Issue confirmed by AWS, and development of a fix in progress.

2023-04-24: Development finished and deployment of the fix in progress.

2023-05-12: Fix deployed globally.

Acknowledgements

We would like to thank the AWS teams involved, especially the Security team for their open communication and helpful responses. I’d also like to thank my coworkers for diving deep when they see unexpected things in the AWS console.

Reach out to us

I talked on June 12 at fwd:cloudsec about how this journey showed the universal benefits of security features, and you can watch that talk on youtube.

The post Spotted: How we discovered Privilege Escalation, missing CloudTrail data and a race condition in AWS Directory Service appeared first on Cloudar.

Sometimes we run into questions that seem simple, but that have a very complex answer. In this blog post we will explain why “Can I migrate between two Amazon Cognito User Pools” is one of those question, and why it usually leads to a new set of questions, like:

• Where do your users exist (do they federate from a different Identity Provider (IdP), or are they in the User Pool?)
• Are you using MFA in Cognito
• Which configuration settings can you change?
• Are you okay with a migration process in the application?
• Will all users sign in during a certain timeframe?
• Do you want to keep your endpoints and identifiers, or are new endpoints okay?

Lets look at why we are asking those questions, and which Cognito components are tricky to deal with.

Configuration

The configuration of a User Pool can be described by the API, or managed in an Infrastructure as Code tool. This makes it easy to recreate with the same settings. However, besides the new endpoints and ids (see below), you will have to update the federation settings (of you are using sign in through a third party).

Eg. If you are federating users from Active Directory (AD) using SAML, you will have to set up a new integration on the AD side as well, so that user can sign in via AD/SAML. This is the same process as setting up this IdP for the first time.

Endpoints and Ids

If you create a new user pool, it will get a new id. This means that the following things will change. Usually this is not a problem, as these are expected changes (if you migrate to a different product, similar things will change).

• The User Pool Id
• The App Client Ids and Secrets (you also create new App Clients)
• The domain used for the hosted UI. Technically you can move it, but this will include downtime. Using a new domain will always be easier (and give you more control about step-by-step updates and rollbacks)
• The User Pool Id and App Client Id are also part of the Tokens that Cognito returns. If you’re verifying them in your application (which you should), or are using a managed integration between an AWS service and Cognito (eg. Api Gateway), you will have to update these too.

Users and Groups

This is the difficult part: while there are APIs to list/describe users and groups, there is no way to export the password (hash) or the MFA (seed) of a user. This means that in the worst case scenario you have to disable MFA and all users will have to reset their password before they can log in again. There are a few solutions, but none of them are ideal.

Problem: You are using MFA in Cognito

This is the worst case scenario for a migration. There is no way to export or set MFA seeds in Cognito, and usually disabling MFA for everyone is not going to be an option

Problem: You are using the sub attribute in your application

The sub attribute is an unique identifier for every user, and migrating users will create a new identifier. To work around this, you could use a different (custom) attribute to safe an identifier in, and update your application to use that. Filling that attribute comes with its own complexities, though.

Option 1: You are using federation / sign in through a third party

You’re in luck! Since the user and their password exist in the other system, you can setup a new federation and users will still be able to sign in (but  make sure you’ve read the “Configuration” section above).

If you want to go this route to make migration (and failover) easier, there are third party IdPs that you can use via SAML/OIDC. Although in some cases it might make more sense to integrate directly with them (either in Cognito Identity Pools or in the application) instead of putting User Pools in between. One reason to keep Cognito, is that you can create multiple App Clients (and thus applications) on one pool (and this can be done with the IaC tool you are already using). Whereas if you integrate directly, you have to configure everything in the external service (and depending on the protocol you are using, this can be hard to debug)

If you are also using groups, you still have to export and recreate those. Take a look at the AWS Solution mentioned in option 2,  and its limitations for an example of doing that.

Option 2: You don’t mind resetting all passwords.

This is also doable, you can export all users, import them in the new pool and make them set a new password to sign in (if this is a limited set of internal users, you could also set a password for them, but you’d want them to pick their own either way)

There is even an AWS solution (read: Open Source software that you have to manage yourself) that can be used to do the export once or on a schedule (eg. to create backups). Make sure to also check the limitations.

Option 3: Migrating on first sign-in

You can add a Lambda Function to the new User Pool (there is example code, but you’d have to manage this yourself). This function will get the username and password, and can check the original pool and write the existing password to the new pool if the password is correct. Docs:

This has some downsides:

• The old User Pool needs to stay around until the migration is complete (you pay for active users, so this does not have to be a cost problem)
• This can take a long time, and potential never finish (if you have users that don’t sign in regularly)

Conclusion

The post Options for migrating between Amazon Cognito User Pools appeared first on Cloudar.

#1 Lacking password management

If the cloud is your kingdom, passwords and credentials are the key to getting into it. Credentials grant you access to all kinds of exploitable data. They are an absolute goldmine for anyone with bad intentions. Research shows that over 50% of users use the same password for almost all of their accounts. That means that theoretically half of the people who can access your cloud infrastructure, do it with the same password they use for their Facebook or Google account. The moment the big players suffer a data leak, hackers can use those passwords to try to get into all of the users’ other accounts, including your organization’s cloud data.

Preventing this data breach starts with setting the right password rules. Think about password length and more complex requirements such as including numbers and symbols. Users will now be obligated to choose a strong password. To avoid the reuse of passwords, you need to be able to remember them all, a seemingly impossible feat. But that’s where a good password manager comes in. So educate your users in good password management, and practice what you preach!

#2 Insufficient encryption

Traffic and data encryption creates trust among approved users. It makes it possible to work efficiently without any security risks. Sending forms, files and data through encrypted channels protects your data, your users and your company. We call this encryption in transit. Encryption is also an essential part of secure data storage. You cannot afford any weak links in the protection of sensitive data. We call this encryption at rest. Be sure to prioritize data and traffic encryption in your AWS cloud setup.

#3 Insider threat

As much as you train and educate your users, sometimes they still may unintentionally cause your cloud security to be at risk. By installing an unauthorized extension to their browser or too many failed login attempts, for example. You need to be able to pick up on unusual network activity like this in order to follow up with the user in question or snuff out an actual security threat. That requires consistent real time monitoring, logging and archiving those logs on encrypted storage. Ensure that you set up a variety of monitoring alerts that notify you of any aberrant behavior.

#4 Phishing

No less than 90% of system breaches start with users opening a phishing email, waich they are shown to do in about 30% of cases. There are two things you can do to get those numbers down. First of all, train your users in recognizing phishing attempts and teach them to report these to your security team. Secondly, protect your AWS root account. You can do this by enabling multi-factor authentication. Also, make sure the root account isn’t linked to any personal or external accounts. The next step would be to use a multiple AWS account strategy. By doing this, not all of your assets will be at risk when one of them is compromised.

#5 Not planning ahead

No matter how well you set up your cloud environment, things will go wrong. Accidents will happen. Unforeseen issues will arise. That’s a normal part of working in evolving technologies. But what it all comes down to at a moment like that is having an adequate incident recovery plan. Essentially, that means you imagine every worst-case scenario beforehand and create a plan for how you will handle it, should it come to that. That way, even if you are faced with a security issue, you will know exactly what to do and be able to minimize your downtime as well as any loss of data, resources and time.

Do you want to exchange ideas about how well-prepared your organization is for cloud related security threats? Get in touch with Cloudar, an official Next Generation AWS Managed Service Provider Partner.

The post The 5 biggest AWS cloud security threats appeared first on Cloudar.