Below is our attempt at a complete answer, from what ESC is to how to choose between commercial, sovereign and hybrid. It ends with the most interesting thing AWS has announced about ESC so far: on Saturday 24 October, it will cut ESC off from the AWS global network on purpose.

What the European Sovereign Cloud actually is

ESC became generally available on 14 January 2026 with one Region, eusc-de-east-1 in Brandenburg, Germany, and more than 90 services. Today, AWS’s Capabilities by Region explorer lists 104 services as available there (with, for example, each RDS engine and FSx variant counted separately), and 886 services and features in total. Three things make it more than “another EU Region”.

It is a separate partition. Next to the commercial cloud (aws), AWS already runs separate partitions for China (aws-cn), GovCloud (aws-us-gov) and isolated US classified Regions. ESC adds aws-eusc, with its own IAM, its own AWS Organizations, its own billing in euro, its own console and its own endpoints under amazonaws.eu. ARNs start with arn:aws-eusc:. Existing accounts do not carry over; you open new ones.

It has its own control plane, in the EU. A dedicated Route 53 whose name servers use only European top-level domains, its own EU trust service provider and root certificate authority, and a dedicated European security operations centre. Customer content stays in the EU, and so does customer-created metadata such as roles, permissions, tags and configurations. In exceptional cases, EU-resident ESC staff have independent access to a replica of the source code needed to maintain the services.

It is run by a European company, with European people. ESC operates through German GmbH entities led by EU citizens, with an advisory board of EU citizens that includes two independent members. Day-to-day operations are carried out exclusively by EU residents located in the EU, and AWS is gradually moving to EU citizens only.

AWS plans to invest more than €7.8 billion in Germany through 2040, and backs the governance with a Sovereign Reference Framework (ESC-SRF) in AWS Artifact and a first set of attestations in March 2026: SOC 2 Type 1 and C5 Type 1 reports covering 69 services, plus seven ISO certifications. Type 1 means control design at a point in time, not operating effectiveness over a period; normal for a young cloud, but relevant if your auditor expects Type 2. Sovereign Local Zones in Brussels, Amsterdam and Lisbon were announced at launch and are still listed as announced, without a date. AWS has not announced a second ESC Region yet, but we expect more to follow. Read everything this post says about a single Region as a snapshot, not a permanent property.

What ESC does not change is who ultimately owns it: Amazon.com, Inc. The ESC Addendum commits AWS to redirect government requests to the customer, notify you, and challenge requests that conflict with EU law. It also says AWS will not move your content or customer-created metadata outside the EU “except as instructed by Customer or as necessary to comply with a valid and binding order of a governmental body”. AWS reports that no enterprise or government content stored outside the US has been disclosed to the US government since it started publishing that figure in 2020. Whether that is enough for you is a legal judgement, and one that has not been tested in court. We are not lawyers; put that question to yours.

How it differs from the other hyperscalers’ sovereign clouds

Every major provider now sells something called sovereign, and the word covers very different architectures. At hyperscale, there are two.

Microsoft describes its Sovereign Public Cloud as “built on Microsoft’s global Azure infrastructure with enhanced sovereignty controls, such as Data Guardian, External Key Management, and tamper-evident access logs”. And in fairness, the separate-cloud model is not an AWS invention: Oracle has run its EU Sovereign Cloud as a separate realm since 2023.

Two kinds of offer are left out of this comparison on purpose. Disconnected and air-gapped clouds run a much smaller service set and solve a different problem. Partner clouds, such as Bleu and Delos Cloud on Microsoft technology or S3NS on Google technology, are a different category: a European company runs the hyperscaler’s software on its own, national-scale infrastructure. The catch is scale: capacity limited to what the partner has built, a core service set rather than the full catalogue, and features that arrive only after the licensor’s release cycle and the partner’s own checks. If your requirement is a European-owned operator, that is where to look. If it is hyperscale behind a sovereign boundary, they are not comparable to ESC.

What sets ESC apart is the combination: the same APIs, tooling and engineering organisation as commercial AWS, behind a partition boundary with its own identity, billing, DNS and PKI, run by EU entities. It is a partition, not a setting. That is its biggest strength and, as we will see, the source of most of its cost.

ESC versus Oracle EU Sovereign Cloud

Oracle’s EU Sovereign Cloud is the closest analogue, so it deserves a direct comparison. Both are separate clouds (a partition at AWS, a realm at Oracle) owned and run by EU legal entities, with support and operations restricted to EU residents. The differences are in how far the independence goes, and how much of it is documented.

Where AWS has an advantage. Operational autonomy and independence from global infrastructure are ESC’s strongest differentiators. AWS explicitly documents that ESC has no critical dependencies on infrastructure outside the EU, including the ability to keep operating if connectivity with the global AWS network is interrupted, and it is about to demonstrate that in public (more on that at the end). AWS also documents independent identity management, billing and metering, EU-based trust services and a dedicated security operations centre. Oracle states that its realm is “completely isolated both physically and logically from other Oracle realms”, but its public documentation does not describe separate identity, billing, trust services or a security operations centre at that level of detail. On governance, ESC’s advisory board includes two independent members who are not Amazon employees, a mechanism on top of the EU-incorporated operating subsidiaries. Oracle documents boards of directors made up of European nationals across its seven EU entities, but does not mention independent members.

Where Oracle has an edge. Two regions, Frankfurt and Madrid, against ESC’s one today, which matters if you need disaster recovery without leaving the sovereign cloud. We expect AWS to add ESC Regions over time, but until it does, that gap is real. A realm that Oracle says supports all its commercial services except some. And three years in the market.

The short version: if your requirement is demonstrable operational independence and governance, ESC documents and tests more. If you need disaster recovery without leaving the sovereign cloud today, Oracle is ahead.

What the analyst quadrants do, and do not, tell you

In Gartner’s 2026 Magic Quadrant for Strategic Cloud Platform Services (1 September 2026), AWS is a Leader for the 16th consecutive year and placed highest on Ability to Execute. Google, also a Leader, is placed furthest on Completeness of Vision.

But a Magic Quadrant positions vendors, not partitions. Nobody gets a separate dot for their sovereign offer. The quadrant tells you about the platform and the engineering organisation behind ESC, which counts for a cloud that is nine months old; it does not assess ESC itself. Nor does it tell you how ESC’s catalogue, ecosystem or sovereignty posture compares with Bleu, S3NS or Oracle. Gartner has no Magic Quadrant for sovereign cloud; its Market Guide for Cloud Infrastructure Sovereign Solutions (June 2026) describes that market without ranking it.

The closest thing to a sovereign ranking is ISG’s 2025 Provider Lens for Sovereign Cloud Infrastructure Services (EU). It evaluated 19 providers and named AWS a Leader for the third year in a row, with the highest score on portfolio attractiveness, crediting a “sovereign-by-design architecture that ensures data residency and regional independence”. Note the timing: the report appeared on 9 January 2026, days before ESC became generally available. It reflects AWS’s sovereignty portfolio and roadmap before ESC went live, not ESC as it runs today, which makes it as much an endorsement of the commercial Regions’ sovereignty controls as of ESC.

Not going to ESC? The commercial Regions are not the non-sovereign option

At the ESC launch, AWS itself said that “most customers are able to meet their requirements using one of the six existing AWS Regions in the EU”. There is evidence behind that.

On 22 September 2026, AWS announced that it is the first cloud provider whose services are approved for NATO RESTRICTED use by all NATO member nations. Spain’s National Cryptologic Centre (CCN) evaluated AWS against D32, NATO’s directive for handling NATO RESTRICTED information in public cloud, with AWS’s Trusted Secure Enclave Sensitive Edition as the baseline. Approved services can now be used at that level in any AWS Region located in a NATO member nation. Two caveats keep this honest. It is a pre-assessed baseline that gives Allies an accelerated path, not an accreditation of your system; that is still delegated to a member nation and/or the NATO Communications and Information Agency. And NATO RESTRICTED is NATO’s lowest classification level. One detail for Belgian readers: Frankfurt, Paris, Milan, Spain, Stockholm and London are in NATO member nations. Ireland, home of eu-west-1 and of many Belgian workloads, is not. Keep that in mind when designing your Region strategy.

The commercial Regions also carry the controls of the AWS Digital Sovereignty Pledge: the Nitro System, designed so that AWS operators cannot access customer content on EC2 instances, reviewed by NCC Group and written into the AWS Service Terms; AWS KMS External Key Store, which keeps key material in an external key manager you operate outside AWS (AWS recommends it only where that is an explicit requirement, because if it goes down, so does everything that depends on it); Control Tower data residency controls, including Region deny per organisational unit; and Dedicated Local Zones for single-customer infrastructure where you need it.

So the realistic question is not “sovereign or not sovereign”. It is which risks you need to mitigate, and whether a partition boundary mitigates one that these controls cannot.

Available is not the same as available

This is where most ESC plans go wrong. A service on the ESC list does not mean the service you use in Frankfurt is there, with the features you use.

AWS’s Capabilities by Region data, available in AWS Builder Center and through the AWS MCP servers, shows four states: Available, a directional launch quarter, Planning (“evaluating launch strategy”) and Not Expanding, which AWS defines as “will not launch in region”. For eusc-de-east-1, at the time of writing:

• Planned with a date: Amazon CloudFront (Q4 2026), Amazon Inspector and AWS Shield Advanced (Q1 2027).
• Planning, no date: AWS Application Migration Service. For a migration partner, that one stings: rehosting into ESC today means VM Import/Export, AWS Elastic Disaster Recovery or AWS DMS instead of MGN.
• Not Expanding: Amazon Detective, Amazon Security Lake, AWS CloudHSM, Amazon Textract, Amazon Q Developer and Amazon Elastic VMware Service, among others.

The subtler gap is inside services that are available. DynamoDB is there; Global Tables are Not Expanding. Route 53 is there; domain registration is at Planning. S3 is there; Multi-Region Access Points are at Planning. EC2 is there; Reserved Instances are not, so EC2 commitments go through Savings Plans. Some of this follows logically from a single-Region partition and should change once ESC gets more Regions. Until then, it is a gap for an architecture designed in Frankfurt.

Bedrock: the service is there, most of the models are not

Amazon Bedrock is available in ESC. Its model catalogue, at the time of writing, is Amazon Nova Pro, Amazon Nova Lite and three Gemma 4 variants served through the OpenAI-compatible bedrock-mantle endpoint. There are no Anthropic Claude models. Nova 2 Lite and open-weight models from Mistral AI and OpenAI were announced in May for “the coming months” and have not arrived. Guardrails is available; Knowledge Bases, Agents and AgentCore are at Planning.

The Gemma 4 launch in September is the most practical AI building block in ESC today, and AWS’s walkthrough is worth reading. Three things stand out:

• Open weights. Gemma 4 is released under Apache 2.0, so you can evaluate, benchmark and fine-tune the model yourself, without depending on a model provider’s roadmap. For a sovereignty discussion, that is an argument in its own right.
• How inference data is handled. AWS describes a zero operator access model, in which “no service operators can access model input or output during inference”, and zero data retention by default (with limited retention for abuse detection on some models). Every request stays in-Region in eusc-de-east-1.
• An OpenAI-compatible endpoint. Existing OpenAI SDK code moves over by changing the base URL and the key:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://bedrock-mantle.eusc-de-east-1.api.amazonwebservices.eu/v1",
    api_key=os.environ["AWS_BEARER_TOKEN_BEDROCK"],  # short-term Bedrock API key, valid up to 12 hours
)

response = client.responses.create(
    model="google.gemma-4-31b-it",
    input="Summarise this incident report in three bullet points.",
    store=False,
)
print(response.output_text)

The three variants cover the usual trade-offs: the 31B dense model for reasoning and coding (256K token context), the 26B-A4B mixture-of-experts model (3.8 billion active parameters per token) for cost-sensitive, high-throughput work, and the compact E2B for latency-sensitive or multimodal classification (128K context).

AWS has not said why, but part of the reason is structural. In Frankfurt, a model like Claude Sonnet 5.5 is not served in-Region at all: it runs through EU geographic cross-Region inference, which spreads requests over the EU Regions, or through global inference. ESC is one Region without cross-Region inference, so every model needs dedicated capacity in Brandenburg before it can appear. The catalogue will grow, but expect it to trail the commercial EU Regions at least until ESC has more than one Region. If your AI roadmap depends on frontier models, that alone may decide where those workloads live (see also When the model goes dark).

The practical rule: write your requirements at feature level, check them against Capabilities by Region and the per-service difference pages of the ESC User Guide, then test. The two sources do not always agree, which tells you how fast this is moving.

The ecosystem is not there yet

ESC has its own AWS Marketplace, with more than 165 solutions by May 2026, but it is a separate catalogue: only SaaS and AMI products are supported, and a SaaS listing must be deployable in ESC and keep its data there. Some vendors have shipped: AWS reports SAP Cloud ERP Private as generally available on ESC, and Sumo Logic launched its security operations platform there in June. Many others have announced intent and not shipped. At the time of writing:

• Snowflake, announced in 2025 as preparing for ESC, does not list an ESC Region on its supported-regions page.
• MongoDB said in 2025 it was preparing for ESC. The Atlas request for eusc-de-east-1 is at “Future Consideration”, with MongoDB saying in February that it is actively evaluating support.
• Trend Micro has announced plans for Vision One on ESC, without a date.
• Datadog’s AWS integration API accepts the partitions aws, aws-cn and aws-us-gov, not aws-eusc.

That is not a criticism of those vendors: building a SaaS control plane in a new partition is real work, and waiting for demand is a reasonable decision. But this gap matters more than most service gaps, for a reason that is easy to miss: your toolchain is inside your sovereignty boundary. ESC keeps your content and your metadata in the EU under EU operations. An observability, CSPM, backup or ITSM tool that reads your ESC accounts from a control plane outside the EU exports exactly what ESC was built to keep in. Before you design an ESC landing zone, list every tool that holds an IAM role in your accounts and check where each one runs.

Commercial, ESC or hybrid: how to decide

Start with the requirement, not the product. Three questions do most of the work.

1. Who requires sovereignty, and what exactly do they require? “Customer content stays in the EU” is met by any EU Region. “Only EU-resident staff operate the systems”, “customer-created metadata stays in the EU” or “the service keeps running when cut off from non-EU infrastructure” are where ESC earns its place. “A European-owned operator” is met by neither ESC nor any other US-parented option.
2. Does the workload fit inside today’s ESC? Services, features, models and every SaaS tool in the path.
3. Which data actually needs it? Data classification, not the application map, should draw the line.

The answer usually lands in one of three shapes.

AWS’s architecture guidance adds a fourth pattern: ESC as a sovereign failover target, with production in a commercial Region and backups, a pilot light or a warm standby in ESC. If your concern is geopolitical or legal disruption rather than day-to-day operation, backups or a tested pilot light are often the cheapest way to buy that exit option.

Whatever the shape, one rule decides whether hybrid works: the boundary has to follow the data. If the data you put in ESC gets copied into commercial for a dashboard, a model prompt or a support ticket, you have paid for the partition and kept the risk.

Hybrid means two clouds, and you pay for both

A hybrid estate is not one AWS environment with an extra Region. It is two environments that share an API style. AWS’s ESC landing zone guidance is explicit: a role in aws cannot assume a role in aws-eusc (or the reverse), and VPC peering, Transit Gateway, AWS RAM, S3 replication, ECR replication, AMI copy and VPC endpoints do not cross the boundary. In practice that means:

• Two organisations, two landing zones, each with its own management account, guardrails, logging and security services.
• Identity configured twice: a second IAM Identity Center instance federated to the same corporate identity provider, and CI/CD with one OIDC trust per partition, or a deployment plane inside ESC.
• Networking through the front door: TLS over the internet, Site-to-Site VPN or a separate Direct Connect connection. A commercial Direct Connect gateway cannot be extended into ESC.
• A separate supply chain: images and AMIs rebuilt or re-signed inside ESC rather than replicated, and certificates per partition.
• Infrastructure as code that knows about partitions. Anything that hardcodes arn:aws: breaks. Cheap to fix early, expensive to fix late:

# CloudFormation: let the template resolve the partition
Resource: !Sub "arn:${AWS::Partition}:s3:::${BucketName}/*"

# Terraform: same idea
data "aws_partition" "current" {}

locals {
  bucket_arn = "arn:${data.aws_partition.current.partition}:s3:::${var.bucket_name}"
}

Service endpoints move to amazonaws.eu, but IAM service principals such as lambda.amazonaws.com stay on amazonaws.com in both partitions.

Then there is the bill, where three costs stack up.

The unit price. ESC list prices are set in euro, at a figure slightly below the dollar price in Frankfurt. Lambda in ESC costs €0.0000164477 per GB-second against a commercial list price of $0.0000166667 in Frankfurt: about 1.3% lower as a number, but in a stronger currency. In dollar terms the premium is the exchange rate times 0.987, minus one: around 8.5% at 1.10 dollars per euro, around 18% at 1.20. A tecRacer comparison at launch, against Frankfurt, found roughly 15% across S3, FSx, EC2, Lambda, RDS and DynamoDB. Compared with Ireland, EC2 comes out higher, because Ireland is cheaper than Frankfurt to begin with. The flip side: commercial Regions can invoice in euro, but their prices are set in dollars and converted at the exchange rate of the invoice date. ESC list prices are set in euro, so the bill carries no currency risk, which some finance teams will value.

The commitment model. No EC2 Reserved Instances; EC2 commitments go through Savings Plans. Check the commitment options per service before you port an RI-based FinOps practice.

The duplication. A second landing zone, a second security tooling stack, a second identity configuration, a support plan in the second partition, data between partitions billed as data transfer out on the sending side, and the people who run it all. For a small ESC footprint, this fixed overhead easily exceeds the premium on the workload itself.

Which means hybrid is not automatically the cheaper option. When the workloads that could stay in commercial are small, or tightly coupled to the ones that must move, running two clouds and the integration between them can cost more than putting everything in ESC and paying the premium on the lot. Run both numbers before you decide to split.

ESC costs a little more per unit and a lot more per boundary. Put as little behind the boundary as your requirement allows, unless the boundary itself costs more than the premium you are trying to avoid.

Where Cloudar fits

This is the part where we talk about ourselves, so read it with that in mind. Cloudar is an ESC launch partner, holds the AWS Digital Sovereignty Competency, delivers Partner-Led Support in the AWS European Sovereign Cloud, and has built landing zones in the aws-eusc partition. That work has taught us that the hard part is rarely the ESC technology. It is keeping the moving parts consistent, so we have organised our work around the four questions customers actually ask:

Start with the Sovereignty Assessment. It is the step most organisations skip, and it decides whether the other three are needed at all. For many organisations the answer will be to stay in the commercial Regions with the right controls enforced. That is a valid outcome, and we will tell you so.

Saturday 24 October: AWS pulls the cable

Claims about operational independence are easy to write and hard to verify, so AWS is going to test one in public. On Saturday 24 October 2026, for several hours, ESC will run without its connection to the AWS Global Network backbone, the private network that normally carries authorised operational data between AWS locations. A dedicated system that handles limited, controlled transfers of operational data will be fully disabled, and traffic moves to ESC’s own internet connectivity from European ISPs. The exercise was called for by ESC’s managing directors and is run by a team made up entirely of EU residents.

What to expect. AWS says service availability inside ESC, other Regions and Direct Connect are not affected, but there may be brief connectivity disruptions at the start and end while routes converge on other networks. If you run cross-partition integrations over the internet or VPN, keep an eye on your monitoring that weekend.

What it proves. AWS designed the exercise to be consistent with the Commission’s Cloud Sovereignty Framework and BSI’s C3A, and will share an update with regulators and customers afterwards. That is the kind of evidence those frameworks ask for, and more than a policy document can offer.

Which brings us back to where we started. ESC is not the answer to every sovereignty question, and it is not just a label either. It is a partition: real operational separation, real gaps, real costs. The question is not whether ESC is sovereign enough. It is which of your data needs a partition, and whether you are prepared to run two clouds to give it one.

Service, feature and model availability in this post was checked against AWS Capabilities by Region, the ESC User Guide and the Amazon Bedrock model documentation on 16 October 2026. ESC is changing quickly; check before you plan.

Gartner, Magic Quadrant for Strategic Cloud Platform Services, 1 September 2026. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organisation and should not be construed as statements of fact. GARTNER and MAGIC QUADRANT are registered trademarks of Gartner, Inc. and/or its affiliates.

ISG Provider Lens, Multi Public Cloud Services, Sovereign Cloud Infrastructure Services (EU), 2025. ISG does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings. ISG research publications consist of the opinions of ISG’s research organisation and should not be construed as statements of fact.

The post The AWS European Sovereign Cloud is a partition, not a setting. Here is how to decide whether you need it. appeared first on Cloudar.

The first one was filed on November 19, 2017. Seven years ago. The ask: let an admin change a customer’s email address in Jira Service Management. Not migrate accounts through a four-step workaround involving Atlassian ID. Just… change an email address. The kind of thing you’d expect a junior developer to ship on a Tuesday afternoon. As of today, it has 779 votes, 447 watchers, and a status of “Future Consideration.” Which, in SaaS-speak, translates roughly to: “we heard you, we filed it, please stop asking.”

The second one landed in January 2019. The request: make Confluence’s alphabetical page sorting persistent, so new pages don’t just pile up at the bottom like uninvited guests. Six years later: 600 votes, 261 watchers, status “Under Consideration.” Progress!

Now, to be fair to Atlassian, they’re not uniquely terrible. Every major SaaS vendor has a graveyard of feature requests exactly like these. Sensible, obvious, clearly wanted by thousands of paying customers. Just… never quite prioritized. Because they have a roadmap. And you’re not on it.

This is the part of the SaaS brochure they don’t show you.

The Pitch vs. The Reality

SaaS vendors are exceptionally good at one thing before you sign: making you feel like you’re about to get exactly what you need. The demos are polished. The slide decks are beautiful. The onboarding is smooth. Eighty percent of your requirements? Covered, on day one.

It’s that remaining twenty percent where things get interesting.

That twenty percent is where your actual workflows live. The edge cases. The things specific to how your organization actually operates. And when you file a support ticket asking about them, you enter a fascinating parallel universe where time moves differently. Features are “on the roadmap.” Updates will come “in a future release.” Your vote has been registered. Thank you for your feedback.

Meanwhile, you’re paying. Every month. For the product as it exists, not as it was promised.

The Numbers Are Telling

The scale of SaaS sprawl is difficult to overstate. According to BetterCloud’s annual State of SaaSOps report, the average number of SaaS applications per company peaked at 130 in 2022 and even after a wave of consolidation, still sits at over 100 today. That’s more than 100 subscriptions to manage, renew, secure, and integrate. For every single organization.

The waste embedded in that sprawl is just as striking. Gartner estimates that approximately 30% of purchased SaaS licenses go unused, what they bluntly call “toxic spend.” BetterCloud puts a price tag on it: companies report wasting an average of more than $135,000 per year on unused software licenses alone. And Gartner projects SaaS spending will continue growing at around 19% annually, reliably outpacing the budgets meant to fund it.

Do the arithmetic on a typical mid-market tool. Fifty users. €50 per user per month. That’s €30,000 per year. Every year. With annual price increases that arrive in your renewal email as a polite fait accompli. After five years, you’ve spent €150,000-plus on software you don’t own, can’t modify, and can’t easily leave and somewhere between a quarter and a third of those licenses have been sitting idle.

Something Changed

Here’s what’s different in 2026: building software got dramatically cheaper and faster. Not incrementally but by an order of magnitude.

Tools like Claude Code and Kiro represent a new category of agentic coding assistants. They don’t just suggest the next line. They can take a requirement, reason through a solution, write the code, test it, catch errors, and iterate, with minimal human supervision. What previously required a team of developers and months of work can now be done by one technically capable person in days.

This isn’t science fiction. It’s happening right now in engineering teams across the world.

And it fundamentally changes the math on one of the oldest questions in IT: should we build or buy?

What You Get When You Build Your Own

When you build a custom application -even a relatively simple internal tool- you build exactly what you need. No more, no less. You control the data model, the workflow, the integrations, and the roadmap. And critically: you decide what gets built next. Not a product manager in Sydney who’s never seen your workflows.

Running on cloud-native infrastructure like AWS means scalability, security, and availability are largely handled by the platform. The operational gap between “something you built” and “something a vendor hosts for you” has narrowed considerably. And the cost gap has flipped.

A custom-built equivalent to that €30,000/year SaaS tool, developed with AI-assisted tooling and running on serverless AWS infrastructure, might cost a fraction of that annually in operational expenses, after a one-time build investment that has also come down dramatically. More importantly: you own the asset. It does exactly what you need. And you’re not waiting seven years for someone to let you update an email address.

A Word of Honesty

Custom software isn’t free of responsibility. Someone has to maintain it, secure it, and evolve it. The SaaS argument, that someone else handles the infrastructure, the uptime and the patches, is not without merit. And for commodity functions like email, video conferencing, or payroll, that argument still wins. These are solved problems. Building your own would be an expensive act of reinvention.

But pairing custom-built applications with managed cloud services gives you the operational coverage of SaaS without the product dependency. You get uptime. You get security. You get to decide what comes next.

So What Should You Actually Do?

Not everything should be custom-built. That would be its own kind of madness.

The question worth asking is simpler: where are your core differentiating workflows? The processes that encode years of operational knowledge. The edge cases that keep showing up in feature request threads, yours and everyone else’s, year after year, status unchanged.

Those are exactly the features that will never quite make it onto a SaaS vendor’s roadmap.

The question is no longer “can we afford to build?” The question, in 2026, is whether you can afford to keep waiting.

JSDCLOUD-5746 would like a word.

The post Dear SaaS Vendor, We’ve Been Waiting Since 2017. appeared first on Cloudar.

Creating and managing secrets is perhaps as old as humans interacting with each other. Yet despite their critical importance, secrets remain one of the most vulnerable aspects of AWS infrastructure today. In our practice as Dev(Sec)Ops Engineers, we see these challenges daily accross our client environments.

To start, the question needs to be asked: “What exactly is a secret?”  In AWS and Cloud environments in general, a secret is anything used to access systems and/or data. These digital keys unlock potentially critical systems and must be protected from unauthorized access at all costs.

As Dev(Sec)Ops Engineers working in AWS, you are responsible for managing an extensive arry of secrets:

• RDS database usernames and passwords
• API keys for AWS services and third-party integrations
• CodeDeploy and deployment system credentials
• KMS encryption keys for data protection
• Private keys for SSL/TLS certificates and secure communications
• EC2 SSH keys and key pairs
• IAM user credentials for developers, QA, and operations teams
• Application service accounts and roles
• Any username/password combinations
• Sensitive configuration data that could aid attackers

The challenge is not just the volume … it’s the complexity of securing these secrets across multiple AWS accounts and other environments while maintaining operational efficiency.

The Hidden Dangers: How Secrets Get Compromised in AWS

 The Configuration File Trap

Storing secrets in configuration files is convenient but dangerous, especially in AWS environments where these files often end up in S3 buckets or EC2 instances. This risk was nicely illustrated in the RSA conference presentation “Red Team vs Blue Team on AWS” by Kolby Allen and Teri Radichel: Link to video

In the video Kolby deployed AWS resources using sample code found easily on the Internet.  Teri, simulating an attacker, conducted a penetration test and discovered a Lambda function’s configuration file stored in a S3 bucket (for convenience) complete with REDS database credentials in plain tekst.

If you must use configuration files in AWS:

• Maintain separate files for development, QA, and production AWS accounts
• NEVER commit these files to source control systems (GitHub, GitLab, Bitbucket)
• Use S3 bucket policies and encryption, but understand this is still suboptimal
• Consider AWS Systems Manager Parameter Store as a minimum improvement

This vulnerability is so important that it’s formally recognized in the MITRE attack framework:  [CWE-200: Exposure of Sensitive Information] (link)

Public Source Control: A Goldmine for AWS Account Takeovers

The most devastating secrets management failures occur when AWS credentials are committed to public repositories. The scale of this problem is staggering.

The people at TruffleHog constructed a live scanner for this:

https://forager.trufflesecurity.com/explore

This scanner continuously detects AWS access keys, secrets keys and other credentials in public Github commits, revealing the massive and constant stream of exposed AWS and other secrets.

This has a widespread impact: TruffleHog researchers discovered approximately 4,500 secrets among the top 1 million websites, many of which were AWS credentials, as detailed in their comprehensive analysis:

https://trufflesecurity.com/blog/4500-of-the-top-1-million-websites-leaked-source-code-secrets

Leaking these secrets can have far reaching financial consequences:

A Reddit user faced a 26.000 $ bill after IAM was compromised to execute crypto miners:

https://www.reddit.com/r/aws/comments/17p3v1e/account_got_hacked_and_get_26000k_bill/

A Developper was billed 14.000 $ on AWS following similar exposure:

https://dev.to/juanmanuelramallo/i-was-billed-for-14k-usd-on-amazon-web-services-17fn

AWS responded on these massive bills and is now actively scanning GitHub respositories through their AWS Credentials Exposed program and automatically disables discovery IAM access keys, but the frequency of these incidents remains alarmingly high as shown by the Trufflehog data.

Internal Systems: The False Security Blanket in AWS

Private repositories and internal systems create a dangerous illusion of security, even within AWS environments. The 2020 Twitter breach perfectly illustrates this vulnerability:

Attackers breached the perimeter, accessed internal Slack channels where developers had stored AWS credentials and other secrets, and used these to compromise infrastructure in a widely publicized incident:

https://www.zdnet.com/article/twitter-says-hackers-accessed-dms-for-36-users-in-last-weeks-hack

Secrets embedded in code even in fully private AWS services, proliferate across AWS services and do appear in:

• CloudWatch logs from Lambda functions and EC2 instances
• S3 bucket access logs
• CloudTrail event data
• Application Load Balancer logs
• Systems Manager Session Manager history

As such they do create additional attack vectors within your AWS environment.

Runtime Exposure: Secrets in AWS Production Workloads

Running AWS applications create numerous opportunities for secret exposure:

• Configuration files within EC2 instances or container images
• Environment variables visible in ECS task definitions or Lambda configurations
• Memory dumps from EC2 instances containing sensitive data
• Application caches in ElastiCache storing credentials
• CloudWatch logs revealing secrets in error messages
• EC2 instance metadata (IMDSv1) exposing IAM credentials
• Unencrypted S3 bucket metadata and tags
• AWS CloudShell command history
• Bash history on EC2 instances
• Container image layers in ECR with embedded secrets

This non-exhaustive list demonstrates how secrets can leak from multiple AWS vectors without proper handling.

What does AWS offer to fight this battle ? AWS Native Secrets Management

Adopt Just-in-Time Secret Retrieval with AWS Services

Core Principle: Store secrets in AWS-native management systems and retrieve them only when needed.

Instead of embedding secrets in configuration files, implement this AWS-secure workflow:

1. Store secrets in AWS Secrets Manager or Systems Manager Parameter Store
2. Retrieve secrets programmatically using AWS SDKs at runtime
3. Use IAM roles and policies for access control
4. Clear secrets from memory when no longer needed

AWS Access Control: Implement least-privilege IAM principles

• Developers cannot access production secrets via IAM policies
• Applications use IAM roles to retrieve only their required secrets
• Cross-account access is controlled via resource-based policies

AWS-Native Secrets Management Solutions

• AWS Systems Manager Parameter Store
• AWS Secrets Manager (Recommended for Production)

As an MSSP, we recommend AWS Secrets Manager as the gold standard for AWS environments.

Why Secrets Manager over Parameter Store?

• Enhanced security controls with fine-grained IAM integration
• Automatic secret rotation for RDS, DocumentDB, and Redshift
• Cross-account access capabilities essential for multi-account AWS strategies
• Comprehensive audit trails via CloudTrail integration
• No CloudFormation exposure risks unlike Parameter Store
• Encryption at rest using AWS KMS by default

As such we recommend the AWS Secrets Manager for Production workloads, the Encrypted Parameter Store can be used as a configuration store to fetch certain parameters.

How to use this in code ?

Python with boto3:

import boto3
import json

# Using IAM role-based authentication (recommended)
client = boto3.client('secretsmanager', region_name='us-east-1')

try:
    secret_response = client.get_secret_value(SecretId='prod/rds/mysql-credentials')
    secret_dict = json.loads(secret_response['SecretString'])
    
    # Use the secret
    db_username = secret_dict['username']
    db_password = secret_dict['password']
    
except ClientError as e:
    # Handle AWS-specific errors
    if e.response['Error']['Code'] == 'DecryptionFailureException':
        # Secrets Manager can't decrypt the protected secret text using the provided KMS key
        raise e
    elif e.response['Error']['Code'] == 'InternalServiceErrorException':
        # An error occurred on the server side
        raise e
    elif e.response['Error']['Code'] == 'InvalidParameterException':
        # Invalid parameter value
        raise e
    elif e.response['Error']['Code'] == 'InvalidRequestException':
        # Parameter value is not valid for the current state of the resource
        raise e
    elif e.response['Error']['Code'] == 'ResourceNotFoundException':
        # Can't find the resource that you asked for
        raise e

Via Infrastructure as Code:

• Cloudformation:
  • https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-secretsmanager-secret.html
• Terraform:
  • https://docs.aws.amazon.com/prescriptive-guidance/latest/secure-sensitive-data-secrets-manager-terraform/using-secrets-manager-and-terraform.html
• AWS CDK:
  • https://docs.aws.amazon.com/secretsmanager/latest/userguide/cdk.html
• Pulumi:
  • https://www.pulumi.com/registry/packages/aws/api-docs/secretsmanager/secret/

AWS Key Management Service (KMS)

AWS KMS provides dedicated encryption key management, solving the “where to store the encryption key” problem for AWS environments. KMS integrates seamlessly with Secrets Manager and most AWS services.

KMS Best Practices for Secrets:

• Use customer-managed KMS keys for production secrets
• Implement key rotation policies
• Use separate KMS keys per environment/account
• Leverage KMS key policies for fine-grained access control

AWS Systems Manager Parameter Store

While we recommend Secrets Manager for sensitive data, **Parameter Store** works well for:

• Non-sensitive configuration data
• Cost-conscious environments (free tier available)
• Simple use cases without rotation requirements

Multi-Account AWS Strategies

For AWS Organizations with multiple accounts (our recommended approach), consider:

Cross-Account Secrets Access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::PROD-ACCOUNT-ID:role/ApplicationRole"
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "secretsmanager:ResourceTag/Environment": "production"
        }
      }
    }
  ]
}

Multi-Region Considerations:

• Replicate critical secrets across AWS regions
• Use AWS Secrets Manager automatic replication
• Consider data residency requirements

AWS Implementation Best Practices from Our MSSP Experience

Security Architecture Considerations

• Design secure AWS deployment pipelines** using CodePipeline, CodeBuild, Github Actions with ci-cd plugins like trufflehog : https://undercodetesting.com/how-to-hunt-for-sensitive-credentials-using-trufflehog
• Implement comprehensive IAM access management with least-privilege principles
• Establish governance policies using AWS Config and AWS Organizations SCPs/RCPs
• Plan for secret rotation using AWS Secrets Manager automation
• Monitor and audit secret access via CloudTrail and CloudWatch

Operational Excellence in AWS

• Automate secret provisioning using AWS Lambda and CloudFormation/Terraform/…
• Implement emergency access procedures via AWS SSO and break-glass roles
• Establish incident response for compromised secrets using AWS Security Hub or other CSPM/CNAPP
• Regular security assessments using AWS Inspector and third-party tools
• Cost optimization by right-sizing Secrets Manager usage vs. Parameter Store

AWS-Specific Monitoring and Alerting

Set up CloudWatch alarms for:

• Unusual Secrets Manager API calls
• Failed secret retrievals
• Cross-account secret access
• KMS key usage anomalies

Conclusion

Effective AWS secrets management is not just about choosing the right AWS service it’s about implementing a comprehensive security strategy that leverages AWS-native capabilities while addressing the entire lifecycle of sensitive data. The examples and breaches discussed here represent real financial and reputational risks that AWS customers face daily.

Key AWS Takeaways:

1. Never store secrets in configuration files, S3 buckets, or source control
2. Use AWS Secrets Manager for production secrets management
3. Implement just-in-time secret retrieval with AWS SDKs
4. Apply least-privilege IAM policies and roles
5. Leverage AWS KMS for encryption key management
6. Plan for multi-account and multi-region secret strategies

The post AWS Secrets Management: Protecting Your Digital Keys in the Cloud appeared first on Cloudar.

Interestingly, Roles Anywhere does not completely solve the problem of not needing to create long-lived credentials, instead it moves (and reduces) the problem from credential management (rotation) to certificate management (distribution).

Setting up a Certificate Authority (CA) that can create certificates can be done with AWS Private Certificate Authority, and there are some options out there to manage distribution (usually as part of a bigger Device Management solution), but wouldn’t it be nice if the government of Belgium would do that for us?

The Belgian eID

In Belgium every citizens gets an identity card. The government makes sure that these get distributed and renewed, and since 2005 they contain a chip that holds (among other things) a certificate that can be used for digital authentication. If we can tie this to Roles Anywhere, we don’t have to worry about certificate management ourselves.

This eID is the size of a bankcard and the chip can be interacted with using off the shelf smart card readers. Most information can be read without any additional authentication – if you have the correct middleware, but to sign something you need to enter a PIN code.

The different certificates

Every eID holds an “authentication certificate” that is tied to the owner of the eID and has been signed by the “Citizens CA”.  The Citizens CA’s certificate is in turn signed by the Belgium Root CA . The Root certificate is self-signed. If we trust the Belgium Root CA, we will be able to validate any authentication certificate. More information about this setup can be found at https://repository.eidpki.belgium.be/#/home

The authentication certificate has two pieces of information that are going to be useful to us:

• The Issuer, showing us that the certificate is indeed issued by the Citizen CA.
• The Subject, containing the name and National Number (a number that uniquely points to one person) of the owner.

AWS IAM Roles Anywhere

Roles Anywhere allows you to use a certificate and its private key to get temporary credentials. You can think of it as an AssumeRole call, but instead of starting from an Access Key, we start from a certificate.

There are a few components we need:

• Configuration in our AWS Account, to indicate which CA we trust, and which roles can be assumed by different Issuer and/or Subjects. This also includes the creation of IAM Roles.
• Software on our computer to interact with the Card Reader so we can prove we have access to the eID and know its PIN.

For our use case we are going to trust the Belgium Root CA, and create a Role that can be assumed with any certificate issued by the Citizen CA. We will than limit the permissions of that IAM Role, so it can only write to a unique prefix in S3.

We can do that because every component of the Subject in our certificate gets mapped to a principal tag on the role session. E.g. if the subject is “C=BE, SN=Doe, GN=John, serialNumber=70010112345, CN=John Doe (Authentication)”, we have the following principal tags, and we can use principal tags in any policy.

• x509Subject/C: BE
• x509Subject/SN: Doe
• x509Subject/GN: John
• x509Subject/serialNumber: 70010112345
• x509Subject/CN: John Doe (Authentication)

Putting it all together

In our AWS Account

We will create the following resources in our AWS Account:

• An S3 Bucket
• A trust anchor, configured to trust the Belgium Root CA certificate that is publicly available via https://crt.eidpki.belgium.be/eid/brca6.crt.  This way we can use any certificate created by the Root CA of any of its subordinate CAs (like the Citizen CA)
• An IAM Role, where we configure the trust policy to only allow usage of the Role if:
  • The principal is the Roles Anywhere service (rolesanywhere.amazonaws.com)
  • The Issuer is Citizen CA (aws:PrincipalTag/x509Issuer/CN must equal “Citizen CA”)
  • The source is our trust anchor (aws:SourceArn equals the ARN of the Root CA trust anchor)
• An IAM Policy on this role that allows PutObject acces to all objects that start with the unique National Number (aws:PrincipalTag/x509Subject/serialNumber)
• a (Roles Anywhere) profile that ties the trust anchor to the Role. We could configure more settings here, like a mapping of fields from the certificate or a session policy, but don’t need to.

Of course we do not have to do this by hand, we create a CloudFormation template for this. It can be found in our sample repository: https://github.com/WeAreCloudar/cloudformation-samples/tree/main/templates/roles-anywhere-eid

On our machine

On our machine, we will need – besides the eID and its PIN (at least on macOS, where I tested this):

• a SmartCard Reader
• the Roles Anywhere Credential Helper (downloadable from the AWS Website)
• The AWS CLI (or any other tool we want to use with our AWS Credentials)

We also need a few pieces of information:

• The ARNs of the trust anchor, profile and role we created
• Our own National Number and the name of the bucket (to test our policy)

Ideally we’d be able to configure the credential helper to read any eID we insert, and this is possible if you install the Belgian eID Middleware using the following command.

aws_signing_helper credential-process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--certificate "pkcs11:model=Belgium%20eID;object=Authentication" \
--pkcs11-lib '/Library/Belgium Identity Card/Pkcs11/beid-pkcs11.bundle/Contents/MacOS/libbeidpkcs11.dylib

However because of an issue with the helper, this only works on macOS if you build the helper yourself from source. Since we do not want to do that, we can plug in our eID and find and identifier using “aws_signing_helper read-certificate-data”. You should get something like

Matching identities
1) bf013dd57aa34f8f9d4e22ab89dd8eb2 "SERIALNUMBER=70010112345,CN=John Doe (Authentication),C=BE,..."
2) 786f7572198a4a8a82ee7ef3f205d7be "SERIALNUMBER=70010112345,CN=John Doe (Signature),C=BE,..."


Copying everything between quotes, allows us to create a new file, selector.json that looks like this:


[
{
"Key": "x509Subject",
"Value": "SERIALNUMBER=70010112345,CN=John Doe (Authentication),C=BE,..."
}
]

and use that to select our specific certificate:

aws_signing_helper credential-process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--cert-selector file://selector.json

Our smart card reader will now prompt us for a PIN. If we enter the right one, we get credentials as output:

{"Version":1,"AccessKeyId":"ASIA...","SecretAccessKey":"COA...","SessionToken":"IQoJ...","Expiration":"2025-03-28T14:16:07Z"}


We can put this command in our ~/.aws/config file, so the AWS CLI will execute it for us, but for testing purposes it’s slightly easier to use the “serve” option.  The AWS Documentation has a longer explanation about how you can use the credential-process: https://docs.aws.amazon.com/rolesanywhere/latest/userguide/credential-helper.html#credential-helper-examples

Testing with s3

In one terminal we can run a credential / metadata server that can be used by the AWS CLI:

~$ aws_signing_helper serve -process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--cert-selector file://selector.json


This gives us
2025/03/28 14:28:12 Local server started on port: 9911
2025/03/28 14:28:12 Make it available to the sdk by running:
2025/03/28 14:28:12 export AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:9911/

We can than use a second terminal to run:

AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:9911/ aws s3 cp example_file s3://$bucket/serialNumber/70010112345/example_file

and get our file in s3:

upload: example_file to s3://.../serialNumber/70010112345/example_file

Conclusion

This experiment shows that being able to use an external device withIAM Roles, is a very powerful feature, because it can remove a lot of tasks related to certificate and credential management. Being able to do this with something I had in my backpocket made it extra cool.

However, the Belgian eID is probably not the best way to go about this in a real scenario:

• I just gave everyone living in Belgium access to my S3 Bucket. I could use a more scoped-down trust policy, but I would eventually run into limitations of trust-policy size.
• I also didn’t tackle Certificate Revocation Lists (CRL). Every time someone loses their wallet, I would need to invalidate certain certificates – this also runs into size limitations
• I already have a good way to give users access to AWS Accounts, using Identity Center or Cognito.
• The pkcs11 support on macOS seems limited, and I would have to install the middleware everywhere,

This does not mean that Roles Anywhere is not a good solution for this kind of cases:

• If you are already putting certificates on machines (e.g using an MDM solution), AWS IAM Roles Anywhere is a great fit
• Instead of relying on the Belgian Government you can buy an hardware key like a yubikey and use it as your certificate source. This is as secure as using the eID (only the machine with the key plugged in will be able to request credentials), and you would be limiting it to the exact serial number(s) in your trust policy anyway – making CRLs mostly moot.

The post Sign in with your eID: Using AWS IAM Roles Anywhere with a SmartCard Reader appeared first on Cloudar.

Cloudar has achieved the AWS Global Security & Compliance Acceleration (GSCA) certification, underscoring our commitment to providing secure, compliant cloud solutions. For AWS users, this milestone brings several benefits, helping them streamline operations, meet compliance requirements, and ensure the highest levels of security.

About the AWS Global Security & Compliance Acceleration Program

AWS designed the GSCA program to support partners in strengthening the security and compliance of cloud environments. Through the GSCA program, we have gained access to advanced training, resources, and AWS experts, enabling us to fine-tune our security capabilities and stay ahead of ever-evolving threats.

This program emphasizes automation, governance, and continuous monitoring, empowering us to proactively address security challenges and scale securely in the cloud.

Enhanced security posture

With this certification, Cloudar can provide customers with ready-to-go security solutions, customized to fit their unique requirements. From secure infrastructure setup to automated incident response, we can ensure your workloads are protected from day one.

By adhering to AWS’s best practices, we can ensure that your infrastructure is secure by design. We leverage tools like AWS Security Hub, Amazon GuardDuty and AWS Config to provide customers with continuous threat detection, rapid incident response, and real-time insights into security events. This means your business can identify and mitigate risks before they escalate.

Regulatory compliance

Compliance is no longer an afterthought. Cloudar uses AWS-native tools to continuously monitor for regulatory compliance and offers pre-built templates to help meet industry standards, reducing the time and effort required to maintain compliance.

Whether your business operates in a heavily regulated industry like healthcare, finance, or government, the GSCA certification ensures that we can help you meet key regulatory requirements such as GDPR, HIPAA, SOC 2, and more.

Faster time-to-market:

By automating security and compliance processes such as patch management, configuration drift detection, and log monitoring, we can significantly reduce the time it takes to implement secure, compliant workloads on AWS.

Cloudar ensures that our customers can maintain a high security posture without the overhead of manual intervention. This leads to lower operational costs and more efficient resource allocation, so you can focus on innovation and growth, while we handle the complexities of cloud security.

Cloudar helps you leverage the power of AWS Security

Cloudar’s approach goes beyond simply implementing AWS’s built-in tools. We work closely with your teams to tailor solutions that fit your specific business needs, ensuring that you get the most out of the AWS cloud while keeping your data secure and compliant.

We design cloud architectures that follow AWS’s security best practices, but are customized to fit the scale, complexity, and compliance needs of your business.

Whether you’re migrating to the cloud or optimizing existing workloads, Cloudar ensures a smooth integration of AWS security services with your current operations.

But our support doesn’t stop after deployment: Cloudar continuously monitors your cloud environment, ensuring you stay compliant and secure as your business grows.

The Cloudar Difference

At Cloudar, we understand that security and compliance aren’t just checkboxes – they’re critical to building trust with your customers and staying ahead in a competitive market.

By achieving the Global Security & Compliance Acceleration on AWS, Cloudar once again proves that we are not just a partner, but a trusted ally in your cloud journey.

We provide peace of mind that your AWS workloads are not only running efficiently but are also secured against threats and aligned with the latest regulatory standards.

If you’d like to learn more about how Cloudar can enhance your cloud security and compliance, or if you’re ready to take your AWS workloads to the next level, don’t hesitate to get in touch with our team today!

The post Achievement Unlocked: Global Security & Compliance Acceleration on AWS appeared first on Cloudar.

At Cloudar, we’ve always believed that security should be at the heart of every cloud strategy. Our commitment to delivering top-notch cloud security solutions has led us to achieve numerous milestones, and today, we’re excited to announce our latest achievement: Cloudar has earned the prestigious AWS Security Competency!

Building on our ISO/IEC 27001 certification since 2018 and our existing AWS competencies, this accomplishment marks a significant milestone in our journey to provide the highest level of security and service to our customers. In this blog post, we’ll dive into what the AWS Security Competency means for us and, more importantly, what benefits it brings to our valued customers.

What is AWS Security Competency?

The AWS Security Competency is not just a fancy title; it’s a recognition of excellence in designing, implementing, and managing security solutions on the Amazon Web Services platform. AWS awards this competency to organizations that demonstrate exceptional proficiency in securing AWS workloads and infrastructure.

Why it matters

Expertise You Can Trust

Achieving the AWS Security Competency is a rigorous process that involves a deep evaluation of an organization’s expertise in various security domains such as identity and access management, data protection, compliance, and more.
This achievement is a seal of approval from Amazon Web Services itself.
It means that Cloudar’s team of experts possesses a comprehensive understanding of AWS security best practices, threat detection, and incident response. It signifies that we have demonstrated proficiency and skill in protecting your cloud infrastructure and data, giving you peace of mind that your assets are in capable hands.

Cutting-Edge Solutions

Our team at Cloudar stays at the forefront of security technology. Achieving the AWS Security Competency ensures that we are equipped with the latest knowledge and tools to keep your AWS environments secure from evolving threats.
Our partnership with Trend Micro played a crucial role in this accomplishment. Trend Micro is a global leader in cloud security, and our collaboration has allowed us to leverage their cutting-edge technology and expertise to enhance our security solutions. Together, we’ve created a formidable team dedicated to safeguarding our customers’ cloud environments.

Dedication to Continuous Improvement

Security is not a static field; it’s an ever-evolving landscape. To maintain this competency, Cloudar continues to invest in training and staying up-to-date with the latest security trends and technologies. We are committed to providing our customers with the most advanced security solutions available.

Proven Track Record

Our attainment of this competency underscores our extensive experience in designing, implementing, and managing secure AWS solutions.
We understand that one-size-fits-all security solutions rarely work. Cloudar’s expertise allows us to tailor security measures to your unique needs, ensuring that your AWS environment is both secure and efficient. We’ve successfully tackled complex security challenges for customers across various industries, consistently delivering exceptional results.

Benefits for our customers

Compliance Assurance

As an ISO/IEC 27001 certified company, Cloudar understands the importance of compliance.
With Cloudar’s AWS Security Competency, you can trust us to help you meet and exceed regulatory compliance requirements. Our deep understanding of AWS services ensures that your cloud infrastructure aligns with industry-specific standards, making audits and certifications a breeze.

Enhanced Security Posture

With our AWS Security Competency and other AWS competencies, you gain access to top-tier security practices and expertise.
We go above and beyond to ensure your AWS environments are fortified against potential threats and vulnerabilities, significantly reducing the risk of data breaches and disruptions.

Cost Savings

Investing in security is an investment in your business’s longevity. By preventing costly breaches and downtime, Cloudar’s expertise can save you substantial financial resources in the long run.

Increased Efficiency

Our tailored security solutions are designed to integrate seamlessly with your existing AWS environment. This ensures that security measures do not impede your operations, allowing for increased efficiency and productivity.

Peace of Mind

In the event of a security incident, Cloudar’s well-practiced incident response teams are ready to spring into action. Our proactive security measures help mitigate threats before they become major issues, minimizing downtime and data loss.
With Cloudar as your trusted AWS Security Competency partner, you can focus on growing your business, knowing that your cloud infrastructure is fortified against threats. Sleep better at night, knowing that experts are on guard.

Conclusion

Achieving the AWS Security Competency is not just an accolade for Cloudar; it’s a testament to our unwavering dedication to providing the highest level of security for our customers’ AWS environments.

We are committed to keeping your data safe, your operations efficient, and your business thriving. Your cloud’s security is our priority, and together, we’ll reach new horizons.

To learn more about how Cloudar can secure your AWS environment and help your business grow securely, contact us today and experience the difference of working with an AWS Security Competency holder!

The post Elevating cloud security to new heights: Cloudar earns AWS Security Competency! appeared first on Cloudar.

As the Internet evolved from a small interconnect between universities where every resource had a public IP address, and resources were managed based on mutual respect and consensus to the Internet as we know it today with multiple threads, risks, and shortage of IP addresses, the direct and public accessibility of all resources could no longer be sustained.

As stated in the security pillar of the Well Architected Framework, the main security principle in AWS Cloud is to create several network layers and defense in depth.

The framework states more specifically: “Do not place resources in public subnets of your VPC unless they absolutely must receive inbound network traffic from public sources” SEC05-BP01 Create network layers.

So as a best practice, cloud engineers strive to deploy their resources in private subnets whenever possible. Because of the confidential nature of these subnets, engineers had to invent ways to manage their resources in these private subnets.

2. How to Use Jump / Bastion Hosts to Manage Your Private Subnets

A solution to manage the infrastructure in your private subnets is to deploy a bastion host. Before 2018, this was the recommended way, and it was well-documented:

For Linux Workloads:
https://aws.amazon.com/solutions/implementations/linux-bastion/

For Windows Workloads:
https://aws.amazon.com/solutions/implementations/rd-gateway/

Although these solutions do solve the problem stated above (how to manage your Linux and/or Windows workloads in their private subnets), these solutions have several drawbacks:

• These solutions still need a public IP address to be associated with the virtual machines and open ports to enable remote access.
• The Linux solution is vulnerable to SSH multiplexing access.
• As with all software that is publicly accessible, they need to be strictly maintained and patched:
  • https://www.bleepingcomputer.com/news/security/remote-desktop-zero-day-bug-allows-attackers-to-hijack-sessions/
  • https://www.microsoft.com/en-us/security/blog/2021/09/02/a-deep-dive-into-the-solarwinds-serv-u-ssh-vulnerability/
• It’s hard to capture the remote access’s metadata and/or full session logs.
• Password management and access control are needed for the management users.

Because IT is all about answering questions, we are looking for an answer to this question: ‘How can I manage my resources in private subnets in a secure, encrypted, and auditable fashion?’ Luckily, AWS came up with two solutions that answer this question.

3. AWS Systems Manager’s New Session Manager

AWS listened to the drawbacks inherent to using bastion/jump hosts and introduced the Session Manager service as a component of AWS Systems Manager. The Session Manager supports both inside AWS and on-premises:

• Windows
• Linux
• MacOS

This blog post will focus on using it with Elastic Compute Cloud (EC2) Linux instances running in AWS.

3.1 Why AWS Session Manager

Session Manager solves the problem mentioned above: to access instances within private subnets that don’t allow ingress connections from the Internet. This is made possible by the Systems Manager (SSM) agent running on the EC2 instances, which pushes traffic to the Session Manager service managed by AWS outside your EC2 environment. The configuration shown in the diagram below provides an example of how it can be configured:

As you can see in the diagram above, the client machine is outside of AWS. It accesses the internal machines over the Internet using valid user credentials for the AWS environment. To access the internal machines, the client can use several methods:

Via the AWS console, you have two-fold access:

• The AWS Systems Manager console includes access to all the Session Manager capabilities for both administrators and end users.

• The Amazon EC2 console allows end users to connect to the EC2 instances for which they have been granted session permissions.

• Via the AWS Command Line Interface (CLI): this requires the installation of the Session Manager Plugin. It is possible to restrict how users launch sessions, so you could choose only to allow them to invoke a connection via the AWS CLI if your users don’t have access to the console or the other way around.

• The Session Manager SDK consists of libraries and sample code that allows application developers to build front-end applications, such as custom shells or self-service portals for internal users that natively use Session Manager to connect to managed nodes. This allows you to integrate Session Manager into your own tooling or automation workflows.

3.2 Configuration

3.2.1 Network Considerations

As shown in the diagram above, the SSM agent actually uses HTTPS to communicate with the SSM endpoint. As such, port 22 for SSH or port 3389 for RDP are no longer required and should not be allowed as ingress traffic. To make this even more secure, we can use VPC endpoints: they use AWS PrivateLink, so traffic between the target EC2 instances and the Session Manager service does not traverse the Internet. To enable this, create these VPC endpoints in your target VPCs, replacing the region accordingly, for example, eu-west-1

Interface endpoints:

• com.amazonaws.[region].ssm

• com.amazonaws.[region].ssmmessages

• com.amazonaws.[region].ec2messages

• com.amazonaws.[region].ec2 [For Windows workloads]

Optional but recommended:

• com.amazonaws.[region].kms = To encrypt logs in CloudWatch

• com.amazonaws.[region].logs = To send logs to CloudWatch

Gateway endpoint:

• com.amazonaws.region.s3

Systems Manager uses this endpoint to update SSM Agent and to perform patching operations. Systems Manager also uses this endpoint for tasks like uploading output logs that you choose to store in S3 buckets, retrieving scripts or other files you store in buckets, and so on.

If you restrict egress networking on the EC2 machines via security groups, you should use the prefix list of the S3 gateway endpoint to send traffic to this gateway endpoint only. Here is an example of security group configuration for a tightly controlled EC2 instance:

As the interface endpoints replace the DNS addresses of the SSM services, the network traffic with the configuration listed above only flows over AWS PrivateLink. You can even further limit access to the S3 buckets that SSM uses by using the structure listed in the AWS documentation: S3 bucket list

3.2.2 EC2 Prerequisites and IAM Configuration

EC2 Prerequisites

The EC2 instance should run the Systems Manager (SSM) agent to support the Systems Manager. The Amazon Linux AMIs support the agent by default, and it can be installed manually on other Linux systems by installing the Systems Manager. The agent allows a lot more than just connecting with Session Manager. More information about its capabilities is available here. Even if you want to use it on a system that is not currently supported, you can always download the open-source code and modify it yourself. However, AWS will not support a modified version of the agent.

IAM Configuration

Each EC2 Instance must have permission to make API calls to the Session Manager Service. All necessary permissions are available in the Amazon-managed IAM policy: AmazonSSMManagedInstanceCore, and this policy should be attached to an Instance profile.

The policy listed above gives you a good baseline, but if you want to lock this down further, you can look up the information here.

Using this method to access the instances does not require any management of SSH keys. Access is only based on IAM policies since everything is managed through the Session Manager.

When configuring Session Manager access for your end-users, you can limit which instances they can access, whether it’s through the command line or the console, and if they are allowed to tunnel SSH or just use the regular Session Manager shell access.
More information about how to configure the IAM policies is available here.

Some AWS policy templates show how to restrict access to specific EC2 instances, but that will probably not be scalable if you really want to implement this for your organization.

Instead, you can refer to the templates available here to see how to configure the policies to allow access based on tags. As shown here, you will also want to ensure you add permissions for users to terminate only the sessions they started. Otherwise, a user may be able to terminate another user’s session.

3.3. Logging

3.3.1. What the AWS CloudTrail Does

Without configuring anything, AWS CloudTrail will collect basic information about sessions. When someone launches a remote access session with Session Manager, SSM will log an event named “StartSession.” This event will include a number of interesting things, such as:

• The username that launched the session
• Whether the user was authenticated with multi-factor authentication (MFA)
• The originating IP address
• The unique ID of the target EC2 instance
• The session ID

The events in CloudTrail are helpful for getting a sense of who’s logging into your instances and when it’s happening. However, it does not provide any information about what they are doing once they’ve established a session.

3.3.2. Session logs

As mentioned in the overview of AWS Session Manager, SSM uses HTTPS to establish sessions, but it is also possible to use it as a tunnel for other protocols, such as SSH or RDP. Be aware that AWS cannot provide full session logs for those connections if you use the tunneling option. The diagram below shows an example configuration for logging full session data either to CloudWatch or S3 when tunneling is not used:

The logs in CloudWatch look like this:

Next to the CloudWatch logs, logging into S3 is also possible.

Please make sure that you attach a policy to your instance profile that allows writing to the S3 bucket you configured to store your session data. The result is that you will see log files written to the specified bucket with the name of the session (the username and a unique string). The logs contain the commands typed, along with their output, which is the same as what is shown in CloudWatch.

3.4. Conclusion

The question: “How can I manage my resources in private subnets in a secure, encrypted, and auditable fashion?” is answered by using the Systems Manager Sessions Manager.

4. Instance Connect and Instance Connect Endpoint

As there are often multiple ways to reach the same goal, AWS delivers a different option to answer the same question: “How can I manage my resources in private subnets in a secure, encrypted, and auditable fashion.” This can also be solved by using Instance Connect with its newest addition: Instance Connect Endpoint

4.1. What Is Instance Connect?

The Instance Connect feature was introduced in 2019 and offers a solution to control SSH access to your instances using AWS Identity and Access Management (IAM) policies and audit connection requests with AWS CloudTrail events. In addition, you can leverage your existing SSH keys or further enhance your security posture by generating one-time use SSH keys each time an authorized user connects.

Instance Connect works with any SSH client, but you can also easily connect to your instances from the EC2 Console.

The question: “How can I manage my resources in private subnets in a secure, encrypted, and auditable fashion?” is not solved by this approach because:

• The machine still needs a public IP address.
• The connection logs cannot be stored in S3 or CloudWatch.

A few months ago, AWS introduced a new Instance Connect Endpoint feature that solves the first drawback listed above.

4.2. Instance Connect Endpoint

With the Endpoint for Instance Connect (EIC) Service, you no longer need a public IP address on your resource or any agent to connect to your resources. The Instance Connect Endpoint Service works in three ways:

• AWS Management Console
• AWS Command Line Interface (AWS CLI)
• SSH Clients like PuTTY or OpenSSH

4.2.1. Instance Connect Endpoint Features

The EIC Endpoint Service is an identity-aware TCP proxy. As described above, it has two modes:

• AWS CLI | SSH CLients
• AWS Console

The AWS CLI client is used to create a secure WebSocket tunnel from your workstation to the endpoint, where authorization happens with your AWS Identity and Access Management (IAM) credentials. When the tunnel is established, point your preferred SSH client at your loopback address (127.0.0.1 or localhost) and connect as usual.

Second, the Console gives you secure and seamless access to resources inside your VPC. Authentication and authorization are evaluated via IAM before traffic reaches the VPC. The diagram below illustrates a user connecting to an AWS EIC.

As shown above, the EIC Endpoints Service provides a high degree of flexibility.

• It doesn’t require your VPC to have direct Internet connectivity
• No agent is needed on the resource(s) you wish to connect to
• It preserves existing workflows by allowing your preferred client software to connect
• IAM and Security Groups can be used to control access

4.2.2. Instance Connect Endpoint Security and Logging

IAM

The beauty of this solution lies in the fact that access can be limited to what the user needs: EIC Endpoints Service follows the important security requirements in terms of the separation of privileges for the control plane and the data plane. An administrator with full EC2 IAM privileges can create and control EIC Endpoints (the control plane). However, the same administrator cannot use those endpoints without also having EC2 Instance Connect IAM privileges (the data plane). The DevOps Engineers who may need to use EIC Endpoint to tunnel into VPC resources do not require control-plane privileges to do so.

CloudTrail

Records of data plane connections include the IAM principal making the request, their source IP address, the requested destination IP address, and the destination port. An example of connecting to the instance from the console:

{
“eventVersion”: “1.08”,
“userIdentity”: {
“type”: “FederatedUser”,
“principalId”: “123456789:Example”,
“arn”: “arn:aws:sts::123456789:ExampleUser”,
“accountId”: “123456789”,
“accessKeyId”: “ExampleKey”,
“sessionContext”: {
“sessionIssuer”: {
“type”: “IAMExample”,
“principalId”: “ExampleID”,
“arn”: “arn:aws:iam::123456789:user/exampleuser”,
“accountId”: “123456789”,
“userName”: “exampleuser”
},
“webIdFederationData”: {},
“attributes”: {
“creationDate”: “2023-07-24T12:24:55Z”,
“mfaAuthenticated”: “true”
}
}
},
“eventTime”: “2023-07-24T13:19:22Z”,
“eventSource”: “ec2-instance-connect.amazonaws.com”,
“eventName”: “SendSSHPublicKey”,
“awsRegion”: “eu-west-1”,
“sourceIPAddress”: “123456789”,
“userAgent”: “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0”,
“requestParameters”: {
“instanceId”: “i-123456789”,
“instanceOSUser”: “ec2-user”,
“sSHPublicKey”: “ssh-ed25519Example\n”
},
“responseElements”: {
“requestId”: “b2a9eeae-7189-4b6c-ac8a-a41e56ea1a42”,
“success”: true
},
“requestID”: “b2a9eeae-7189-4b6c-ac8a-a41e56ea1a42”,
“eventID”: “161427f7-8f1e-459a-83f1-2f54aca879ca”,
“readOnly”: false,
“eventType”: “AwsApiCall”,
“managementEvent”: true,
“recipientAccountId”: “123456789”,
“eventCategory”: “Management”,
“tlsDetails”: {
“tlsVersion”: “TLSv1.3”,
“cipherSuite”: “TLS_AES_128_GCM_SHA256”,
“clientProvidedHostHeader”: “ec2-instance-connect.eu-west-1.amazonaws.com”
},
“sessionCredentialFromConsole”: “true”
}

4.3. Connection Examples

4.3.1. Via the Console

This gives you:

As you can see, the console does not login via SSM-user but via EC2-user and with a private IP.

4.3.2 Via the AWS CLI

To connect via the AWS CLI, make sure that you have the latest AWS CLI installed. Then you can use the new ec2-instance-connect ssh command from the AWS CLI. With this new command, AWS generates ephemeral keys for you to connect to your Instance.

Note that this command requires using the OpenSSH client and the latest version of the AWS CLI. You need IAM permissions, as detailed here, to use this command and connect. You can find an example below:

4.3.3 Via SSH Clients

Create your own private key:

ssh-keygen -t rsa -f mynew_key

Use the following AWS CLI command to authorize the user and push the public key to the Instance using the send-ssh-public-key command. To support this, you need the latest version of the AWS CLI.

aws ec2-instance-connect send-ssh-public-key –region eu-west-1 –instance-id i-0123456789example –availability-zone eu-west-1a –instance-os-user ec2-user –ssh-public-key file://mynew_key.pub

This gives back:

{
“RequestId”: “9c4bf3df-799d-4f40-9e2d-cbc3ed3bbe08”,
“Success”: true
}

After authentication, the public key is made available to the Instance through the Instance metadata for 60 seconds. During this time, connect to the Instance using the associated private key:

ssh ec2-user@[INSTANCE] \
-i [SSH-KEY] \
-o ProxyCommand=’aws ec2-instance-connect open-tunnel \
–instance-id %h’

This gives back:

With this method, plain SSH clients can still be used via short-lived keys.

4.3.4 Connecting to the Windows Machines

How to connect to Windows machines is similar:

aws ec2-instance-connect open-tunnel \
–instance-id i-0123456789example \
–remote-port 3389 \
–local-port any-port

In your RDP Client, set it up like this, mark that the –local-port is set to 5555

4.4 To Sum Up Instance Connect

As shown in the blog post above, Instance Connect allows you to connect to your private resources privately and securely without needing long-lived SSH keys and/or bastion hosts.
The drawback compared to AWS Sessions Manager is that you cannot log session logs towards Amazon S3 or Amazon CloudWatch. The benefit is that you don’t need to install custom agents on your machines, which makes this solution the preferred solution for older Instances and appliances running in your AWS environment.

5. Final Conclusion

AWS provides various methods to connect to Instances, including SSH, RDP, AWS Systems Manager Session Manager, EC2 Instance Connect and Endpoint, bastion hosts, and VPN. The choice of method depends on the operating system, security requirements, network configuration, and personal preference. By leveraging these connection methods, you can securely access and manage your AWS Instances based on your specific use cases and requirements.

What questions do you have after reading this blog post? We would love to answer them!

The post Managing Instances in Private Subnets with Session Manager or Instance Connect appeared first on Cloudar.

#1 Develop a comprehensive strategy

Before deploying your applications on AWS, it is essential to establish a well-defined cybersecurity strategy. This strategy should include conducting a risk assessment, performing threat modelling, and devising a mitigation plan. Identify potential risks and vulnerabilities specific to your cloud infrastructure and implement measures to address them. This will ensure compliance with industry standards and regulations while proactively preventing cyberattacks.

#2 Become familiar with the AWS Well-Architected Framework

AWS offers abundant resources to help organizations enhance their security posture. The AWS Well-Architected Framework provides guidance for designing and operating secure and cost-effective systems in the cloud. The Framework encompasses five key pillars: operational excellence, security, reliability, performance efficiency, and cost optimization. Understanding and leveraging the Framework is key to safeguarding applications against potential cyberthreats.

#3 Implement robust security controls

Taking proactive measures to impose strong cloud security controls is crucial for organizations to secure their AWS cloud infrastructure effectively. As the responsibility falls on you to protect your cloud workloads, consider implementing security controls to minimize the risk of security breaches, including clearly defining user roles, conducting privilege audits, enforcing a strong password policy, and using multi-factor authentication (MFA). Consistent enforcement and adherence to these security controls throughout your organization are essential.

#4 Ensure easy access to security policies

Accessibility to your AWS security policies is key to promoting a culture of security within your organization. These policies should encompass access control, network security, encryption, and incident response guidelines. Making policies accessible helps everyone in your organization understand the importance of security and act in accordance with your policies.

#5 Safeguard data with encryption

Encrypting your data prevents unauthorized access and ensures its confidentiality, even if intercepted or stolen. Encryption is often mandatory for regulatory compliance. AWS provides various encryption options, including server-side encryption, client-side encryption, and transit encryption. Familiarize yourself with these options and choose the appropriate method to secure your sensitive data within your cloud environment.

#6 Back up data consistently

Regular data backups ensure data integrity and availability. In the event of cyberattacks, hardware failures, or accidental deletions, having up-to-date backups provides peace of mind and enables data recovery. The frequency of backups should be based on the criticality and volatility of your data. Regulatory requirements and business needs will determine how long the data will be retained. AWS offers various backup options, such as Amazon EBS, Amazon S3, and AWS Backup. By leveraging these options, you can store your backups in multiple regions to ensure constant protection.

#7 Stay up to date

AWS regularly releases security patches, bug fixes, and updates to address vulnerabilities and counter new threats. Keeping your AWS systems up to date maintains the security of your environment and protects against potential attacks. Take advantage of AWS’s automatic update features, such as Amazon Inspector and AWS Systems Manager, to automate patching and ensure your systems are always current. Configure notifications to receive alerts about new updates, enabling prompt action to stay ahead of security risks.

Want to optimally prepare your organization for cloud-related security threats? Get in touch with Cloudar, an official Next Generation AWS Managed Service Provider Partner.

The post Get it covered: 7 essential practices for AWS security appeared first on Cloudar.

The findings

When you configure an AWS service to use an IAM role on your behalf, you need to have the iam:PassRole permission in addition to the permissions required to configure and/or use the service.

Due to the bug we reported, AWS Directory Service didn’t verify the presence of those permissions when assigning users or groups to an existing IAM role. This allowed users with the  ds:EnableRoleAccess permission to assign users to any role in their account that has a trust relationship with AWS Directory service.

Additionally, we found some actions that were not logged to CloudTrail and a validation that’s only done client-side

The discovery

Cloudar is an AWS Managed Service Provider Partner, and as such, we work together with our clients to configure their environments. One of our clients alerted us of a strange behavior in the AWS Console for AWS Directory Service, which prompted us to take a deeper look behind the scenes of how that service works.

Step 1: Missing CloudTrail Data

We had configured AWS Directory Service for AWS Management Console access for our client, and it was working properly for them. However, when looking in the console, the feature appeared to be disabled.

Usually, we would investigate CloudTrail logs to see what disabled this, but at the time, we didn’t see an entry in CloudTrail that showed the enabling or disabling of access. When we tested what happened when changing the status ourselves, we also did not see those actions recorded in CloudTrail.

We’ve reported this to AWS, and enabling or disabling console access does now create events in CloudTrail (note: **OMITTED** is part of the CloudTrail event):

{
[…]
    "eventSource": "ds.amazonaws.com",
    "requestParameters": {
        "directoryId": "d-123456789a",
        "applicationId": "***OMITTED***",
        "serviceAccountUserName": "***OMITTED***",
        "serviceAccountPassword": "HIDDEN_DUE_TO_SECURITY_REASONS"
    },
    "responseElements": null,
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "sessionCredentialFromConsole": "true"
}

{
[…]
    "eventSource": "ds.amazonaws.com",
    "eventName": "UnauthorizeApplication",
    "requestParameters": {
        "directoryId": "d-123456789a",
        "applicationId": "***OMITTED***"
    },
    "responseElements": null,
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "sessionCredentialFromConsole": "true"
}

Step 2: Looking at what the console does

Instead of looking through CloudTrail logs, we opened the development console in our browser. There we saw that it was sending requests to an undocumented API when enabling console access. There is no public API for enabling console access or assigning user to roles within Directory Service, so this wasn’t unexpected.

What was unexpected was that when enabling or disabling console access, there were two calls to the internal API. One to (un)authorize an application called “AWSManagementConsole” and one to (un)authorize an application called “Directory Service console”.

Assuming there was a consistency problem, we investigated what happened if we only unauthorized one of them by sending requests to the undocumented API from the command line.  When we unauthorized the “AWSManagementConsole” and authorized “Directory Service console”, we were able to reproduce our customer’s issue where the console would show “disabled”, but the sign-in would still work.

By the time we sent our report to AWS, the double API call didn’t happen anymore (only “AWSManagementConsole” is used today), so this seemed to have been a temporary issue. We still reported this behavior, indicating that we could no longer reproduce.

Step 3: Finding a race condition

Since this wasn’t an easy bug to uncover, we had to read a lot of documentation to deeply understand what Directory Service was doing. While doing that, we noted an interesting sentence: “If any IAM roles have been assigned to users or groups in the directory, the Disable button may be unavailable.”

The word “may” seemed very ambiguous, especially after finding an issue where a network disconnection could lead to an unexpected state. We looked a bit further and realized that the expected behavior is that you can’t disable the integration if you still have users and/or groups assigned to roles.

However, that check is done completely in the browser, and appears to be based on the state when the tab was loaded. This means you can disable the console access while still having users and/or groups assigned. You could do this by calling the same (undocumented) API that the console uses, or in the browser by 1) opening the console while there are no assignments 2) using a second tab to assign users and/or groups to a role 3) using the first tab to disable console access.

We’ve reported this issue to AWS, but don’t consider it a vulnerability as the assignments are not usable if the Console access is disabled.

Step 4: Privilege escalation

Having discovered two APIs that did some part of their logic client-side, and having experimented with the role-assignment, we looked at other validations that should happen when you assign a Role to a user in a directory.  To assign a specific role to a user or group:

• you should have ds:EnableRoleAccess permission for the Directory Service domain;
• the role you want to pass should have a trust policy that allows use in AWS Directory Service;
• the role should be in the same AWS account;
• you should have iam:PassRole permissions on the role.

When we tested these checks, we noticed that we could assign a role (within the same account), even if we didn’t have the right iam:PassRole permission. In certain setups, this could mean privilege escalation.

• Imagine you have two people, Alice and Bob, managing a domain, and each of them is responsible for assigning users to specific IAM roles. Alice assigns people to Role A and Bob assigns people to Role B. They could abuse the missing iam:PassRole check to assign people to a role they are not responsible for. For example, Alice assigns herself (or someone else) to Role B.
• Another possible scenario would be when a user has the required permissions to create and manage a domain in AWS Directory Service, but no IAM permissions (e.g., with the PowerUserAccess managed policy). If there is already a role that has a trust relationship with AWS Directory Service, that user could create their own domain and assume that existing role.

This issue has been completely fixed.

Recommended actions

Taking advantage of this vulnerability required a specific configuration and a high level of permissions within Directory Service. AWS customers who were relying on iam:PassRole permissions for access control before 2023-05-12 should review their directory for misconfiguration and their CloudTrail logs for unexpected AssumeRole calls.

Timeline

2023-04-07: Issue found and reported to AWS.

2023-04-07: Investigation started by AWS.

2023-04-13: Issue confirmed by AWS, and development of a fix in progress.

2023-04-24: Development finished and deployment of the fix in progress.

2023-05-12: Fix deployed globally.

Acknowledgements

We would like to thank the AWS teams involved, especially the Security team for their open communication and helpful responses. I’d also like to thank my coworkers for diving deep when they see unexpected things in the AWS console.

Reach out to us

I talked on June 12 at fwd:cloudsec about how this journey showed the universal benefits of security features, and you can watch that talk on youtube.

The post Spotted: How we discovered Privilege Escalation, missing CloudTrail data and a race condition in AWS Directory Service appeared first on Cloudar.

Sometimes we run into questions that seem simple, but that have a very complex answer. In this blog post we will explain why “Can I migrate between two Amazon Cognito User Pools” is one of those question, and why it usually leads to a new set of questions, like:

• Where do your users exist (do they federate from a different Identity Provider (IdP), or are they in the User Pool?)
• Are you using MFA in Cognito
• Which configuration settings can you change?
• Are you okay with a migration process in the application?
• Will all users sign in during a certain timeframe?
• Do you want to keep your endpoints and identifiers, or are new endpoints okay?

Lets look at why we are asking those questions, and which Cognito components are tricky to deal with.

Configuration

The configuration of a User Pool can be described by the API, or managed in an Infrastructure as Code tool. This makes it easy to recreate with the same settings. However, besides the new endpoints and ids (see below), you will have to update the federation settings (of you are using sign in through a third party).

Eg. If you are federating users from Active Directory (AD) using SAML, you will have to set up a new integration on the AD side as well, so that user can sign in via AD/SAML. This is the same process as setting up this IdP for the first time.

Endpoints and Ids

If you create a new user pool, it will get a new id. This means that the following things will change. Usually this is not a problem, as these are expected changes (if you migrate to a different product, similar things will change).

• The User Pool Id
• The App Client Ids and Secrets (you also create new App Clients)
• The domain used for the hosted UI. Technically you can move it, but this will include downtime. Using a new domain will always be easier (and give you more control about step-by-step updates and rollbacks)
• The User Pool Id and App Client Id are also part of the Tokens that Cognito returns. If you’re verifying them in your application (which you should), or are using a managed integration between an AWS service and Cognito (eg. Api Gateway), you will have to update these too.

Users and Groups

This is the difficult part: while there are APIs to list/describe users and groups, there is no way to export the password (hash) or the MFA (seed) of a user. This means that in the worst case scenario you have to disable MFA and all users will have to reset their password before they can log in again. There are a few solutions, but none of them are ideal.

Problem: You are using MFA in Cognito

This is the worst case scenario for a migration. There is no way to export or set MFA seeds in Cognito, and usually disabling MFA for everyone is not going to be an option

Problem: You are using the sub attribute in your application

The sub attribute is an unique identifier for every user, and migrating users will create a new identifier. To work around this, you could use a different (custom) attribute to safe an identifier in, and update your application to use that. Filling that attribute comes with its own complexities, though.

Option 1: You are using federation / sign in through a third party

You’re in luck! Since the user and their password exist in the other system, you can setup a new federation and users will still be able to sign in (but  make sure you’ve read the “Configuration” section above).

If you want to go this route to make migration (and failover) easier, there are third party IdPs that you can use via SAML/OIDC. Although in some cases it might make more sense to integrate directly with them (either in Cognito Identity Pools or in the application) instead of putting User Pools in between. One reason to keep Cognito, is that you can create multiple App Clients (and thus applications) on one pool (and this can be done with the IaC tool you are already using). Whereas if you integrate directly, you have to configure everything in the external service (and depending on the protocol you are using, this can be hard to debug)

If you are also using groups, you still have to export and recreate those. Take a look at the AWS Solution mentioned in option 2,  and its limitations for an example of doing that.

Option 2: You don’t mind resetting all passwords.

This is also doable, you can export all users, import them in the new pool and make them set a new password to sign in (if this is a limited set of internal users, you could also set a password for them, but you’d want them to pick their own either way)

There is even an AWS solution (read: Open Source software that you have to manage yourself) that can be used to do the export once or on a schedule (eg. to create backups). Make sure to also check the limitations.

Option 3: Migrating on first sign-in

You can add a Lambda Function to the new User Pool (there is example code, but you’d have to manage this yourself). This function will get the username and password, and can check the original pool and write the existing password to the new pool if the password is correct. Docs:

This has some downsides:

• The old User Pool needs to stay around until the migration is complete (you pay for active users, so this does not have to be a cost problem)
• This can take a long time, and potential never finish (if you have users that don’t sign in regularly)

Conclusion

The post Options for migrating between Amazon Cognito User Pools appeared first on Cloudar.