Below is our attempt at a complete answer, from what ESC is to how to choose between commercial, sovereign and hybrid. It ends with the most interesting thing AWS has announced about ESC so far: on Saturday 24 October, it will cut ESC off from the AWS global network on purpose.

What the European Sovereign Cloud actually is

ESC became generally available on 14 January 2026 with one Region, eusc-de-east-1 in Brandenburg, Germany, and more than 90 services. Today, AWS’s Capabilities by Region explorer lists 104 services as available there (with, for example, each RDS engine and FSx variant counted separately), and 886 services and features in total. Three things make it more than “another EU Region”.

It is a separate partition. Next to the commercial cloud (aws), AWS already runs separate partitions for China (aws-cn), GovCloud (aws-us-gov) and isolated US classified Regions. ESC adds aws-eusc, with its own IAM, its own AWS Organizations, its own billing in euro, its own console and its own endpoints under amazonaws.eu. ARNs start with arn:aws-eusc:. Existing accounts do not carry over; you open new ones.

It has its own control plane, in the EU. A dedicated Route 53 whose name servers use only European top-level domains, its own EU trust service provider and root certificate authority, and a dedicated European security operations centre. Customer content stays in the EU, and so does customer-created metadata such as roles, permissions, tags and configurations. In exceptional cases, EU-resident ESC staff have independent access to a replica of the source code needed to maintain the services.

It is run by a European company, with European people. ESC operates through German GmbH entities led by EU citizens, with an advisory board of EU citizens that includes two independent members. Day-to-day operations are carried out exclusively by EU residents located in the EU, and AWS is gradually moving to EU citizens only.

AWS plans to invest more than €7.8 billion in Germany through 2040, and backs the governance with a Sovereign Reference Framework (ESC-SRF) in AWS Artifact and a first set of attestations in March 2026: SOC 2 Type 1 and C5 Type 1 reports covering 69 services, plus seven ISO certifications. Type 1 means control design at a point in time, not operating effectiveness over a period; normal for a young cloud, but relevant if your auditor expects Type 2. Sovereign Local Zones in Brussels, Amsterdam and Lisbon were announced at launch and are still listed as announced, without a date. AWS has not announced a second ESC Region yet, but we expect more to follow. Read everything this post says about a single Region as a snapshot, not a permanent property.

What ESC does not change is who ultimately owns it: Amazon.com, Inc. The ESC Addendum commits AWS to redirect government requests to the customer, notify you, and challenge requests that conflict with EU law. It also says AWS will not move your content or customer-created metadata outside the EU “except as instructed by Customer or as necessary to comply with a valid and binding order of a governmental body”. AWS reports that no enterprise or government content stored outside the US has been disclosed to the US government since it started publishing that figure in 2020. Whether that is enough for you is a legal judgement, and one that has not been tested in court. We are not lawyers; put that question to yours.

How it differs from the other hyperscalers’ sovereign clouds

Every major provider now sells something called sovereign, and the word covers very different architectures. At hyperscale, there are two.

Microsoft describes its Sovereign Public Cloud as “built on Microsoft’s global Azure infrastructure with enhanced sovereignty controls, such as Data Guardian, External Key Management, and tamper-evident access logs”. And in fairness, the separate-cloud model is not an AWS invention: Oracle has run its EU Sovereign Cloud as a separate realm since 2023.

Two kinds of offer are left out of this comparison on purpose. Disconnected and air-gapped clouds run a much smaller service set and solve a different problem. Partner clouds, such as Bleu and Delos Cloud on Microsoft technology or S3NS on Google technology, are a different category: a European company runs the hyperscaler’s software on its own, national-scale infrastructure. The catch is scale: capacity limited to what the partner has built, a core service set rather than the full catalogue, and features that arrive only after the licensor’s release cycle and the partner’s own checks. If your requirement is a European-owned operator, that is where to look. If it is hyperscale behind a sovereign boundary, they are not comparable to ESC.

What sets ESC apart is the combination: the same APIs, tooling and engineering organisation as commercial AWS, behind a partition boundary with its own identity, billing, DNS and PKI, run by EU entities. It is a partition, not a setting. That is its biggest strength and, as we will see, the source of most of its cost.

ESC versus Oracle EU Sovereign Cloud

Oracle’s EU Sovereign Cloud is the closest analogue, so it deserves a direct comparison. Both are separate clouds (a partition at AWS, a realm at Oracle) owned and run by EU legal entities, with support and operations restricted to EU residents. The differences are in how far the independence goes, and how much of it is documented.

Where AWS has an advantage. Operational autonomy and independence from global infrastructure are ESC’s strongest differentiators. AWS explicitly documents that ESC has no critical dependencies on infrastructure outside the EU, including the ability to keep operating if connectivity with the global AWS network is interrupted, and it is about to demonstrate that in public (more on that at the end). AWS also documents independent identity management, billing and metering, EU-based trust services and a dedicated security operations centre. Oracle states that its realm is “completely isolated both physically and logically from other Oracle realms”, but its public documentation does not describe separate identity, billing, trust services or a security operations centre at that level of detail. On governance, ESC’s advisory board includes two independent members who are not Amazon employees, a mechanism on top of the EU-incorporated operating subsidiaries. Oracle documents boards of directors made up of European nationals across its seven EU entities, but does not mention independent members.

Where Oracle has an edge. Two regions, Frankfurt and Madrid, against ESC’s one today, which matters if you need disaster recovery without leaving the sovereign cloud. We expect AWS to add ESC Regions over time, but until it does, that gap is real. A realm that Oracle says supports all its commercial services except some. And three years in the market.

The short version: if your requirement is demonstrable operational independence and governance, ESC documents and tests more. If you need disaster recovery without leaving the sovereign cloud today, Oracle is ahead.

What the analyst quadrants do, and do not, tell you

In Gartner’s 2026 Magic Quadrant for Strategic Cloud Platform Services (1 September 2026), AWS is a Leader for the 16th consecutive year and placed highest on Ability to Execute. Google, also a Leader, is placed furthest on Completeness of Vision.

But a Magic Quadrant positions vendors, not partitions. Nobody gets a separate dot for their sovereign offer. The quadrant tells you about the platform and the engineering organisation behind ESC, which counts for a cloud that is nine months old; it does not assess ESC itself. Nor does it tell you how ESC’s catalogue, ecosystem or sovereignty posture compares with Bleu, S3NS or Oracle. Gartner has no Magic Quadrant for sovereign cloud; its Market Guide for Cloud Infrastructure Sovereign Solutions (June 2026) describes that market without ranking it.

The closest thing to a sovereign ranking is ISG’s 2025 Provider Lens for Sovereign Cloud Infrastructure Services (EU). It evaluated 19 providers and named AWS a Leader for the third year in a row, with the highest score on portfolio attractiveness, crediting a “sovereign-by-design architecture that ensures data residency and regional independence”. Note the timing: the report appeared on 9 January 2026, days before ESC became generally available. It reflects AWS’s sovereignty portfolio and roadmap before ESC went live, not ESC as it runs today, which makes it as much an endorsement of the commercial Regions’ sovereignty controls as of ESC.

Not going to ESC? The commercial Regions are not the non-sovereign option

At the ESC launch, AWS itself said that “most customers are able to meet their requirements using one of the six existing AWS Regions in the EU”. There is evidence behind that.

On 22 September 2026, AWS announced that it is the first cloud provider whose services are approved for NATO RESTRICTED use by all NATO member nations. Spain’s National Cryptologic Centre (CCN) evaluated AWS against D32, NATO’s directive for handling NATO RESTRICTED information in public cloud, with AWS’s Trusted Secure Enclave Sensitive Edition as the baseline. Approved services can now be used at that level in any AWS Region located in a NATO member nation. Two caveats keep this honest. It is a pre-assessed baseline that gives Allies an accelerated path, not an accreditation of your system; that is still delegated to a member nation and/or the NATO Communications and Information Agency. And NATO RESTRICTED is NATO’s lowest classification level. One detail for Belgian readers: Frankfurt, Paris, Milan, Spain, Stockholm and London are in NATO member nations. Ireland, home of eu-west-1 and of many Belgian workloads, is not. Keep that in mind when designing your Region strategy.

The commercial Regions also carry the controls of the AWS Digital Sovereignty Pledge: the Nitro System, designed so that AWS operators cannot access customer content on EC2 instances, reviewed by NCC Group and written into the AWS Service Terms; AWS KMS External Key Store, which keeps key material in an external key manager you operate outside AWS (AWS recommends it only where that is an explicit requirement, because if it goes down, so does everything that depends on it); Control Tower data residency controls, including Region deny per organisational unit; and Dedicated Local Zones for single-customer infrastructure where you need it.

So the realistic question is not “sovereign or not sovereign”. It is which risks you need to mitigate, and whether a partition boundary mitigates one that these controls cannot.

Available is not the same as available

This is where most ESC plans go wrong. A service on the ESC list does not mean the service you use in Frankfurt is there, with the features you use.

AWS’s Capabilities by Region data, available in AWS Builder Center and through the AWS MCP servers, shows four states: Available, a directional launch quarter, Planning (“evaluating launch strategy”) and Not Expanding, which AWS defines as “will not launch in region”. For eusc-de-east-1, at the time of writing:

• Planned with a date: Amazon CloudFront (Q4 2026), Amazon Inspector and AWS Shield Advanced (Q1 2027).
• Planning, no date: AWS Application Migration Service. For a migration partner, that one stings: rehosting into ESC today means VM Import/Export, AWS Elastic Disaster Recovery or AWS DMS instead of MGN.
• Not Expanding: Amazon Detective, Amazon Security Lake, AWS CloudHSM, Amazon Textract, Amazon Q Developer and Amazon Elastic VMware Service, among others.

The subtler gap is inside services that are available. DynamoDB is there; Global Tables are Not Expanding. Route 53 is there; domain registration is at Planning. S3 is there; Multi-Region Access Points are at Planning. EC2 is there; Reserved Instances are not, so EC2 commitments go through Savings Plans. Some of this follows logically from a single-Region partition and should change once ESC gets more Regions. Until then, it is a gap for an architecture designed in Frankfurt.

Bedrock: the service is there, most of the models are not

Amazon Bedrock is available in ESC. Its model catalogue, at the time of writing, is Amazon Nova Pro, Amazon Nova Lite and three Gemma 4 variants served through the OpenAI-compatible bedrock-mantle endpoint. There are no Anthropic Claude models. Nova 2 Lite and open-weight models from Mistral AI and OpenAI were announced in May for “the coming months” and have not arrived. Guardrails is available; Knowledge Bases, Agents and AgentCore are at Planning.

The Gemma 4 launch in September is the most practical AI building block in ESC today, and AWS’s walkthrough is worth reading. Three things stand out:

• Open weights. Gemma 4 is released under Apache 2.0, so you can evaluate, benchmark and fine-tune the model yourself, without depending on a model provider’s roadmap. For a sovereignty discussion, that is an argument in its own right.
• How inference data is handled. AWS describes a zero operator access model, in which “no service operators can access model input or output during inference”, and zero data retention by default (with limited retention for abuse detection on some models). Every request stays in-Region in eusc-de-east-1.
• An OpenAI-compatible endpoint. Existing OpenAI SDK code moves over by changing the base URL and the key:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://bedrock-mantle.eusc-de-east-1.api.amazonwebservices.eu/v1",
    api_key=os.environ["AWS_BEARER_TOKEN_BEDROCK"],  # short-term Bedrock API key, valid up to 12 hours
)

response = client.responses.create(
    model="google.gemma-4-31b-it",
    input="Summarise this incident report in three bullet points.",
    store=False,
)
print(response.output_text)

The three variants cover the usual trade-offs: the 31B dense model for reasoning and coding (256K token context), the 26B-A4B mixture-of-experts model (3.8 billion active parameters per token) for cost-sensitive, high-throughput work, and the compact E2B for latency-sensitive or multimodal classification (128K context).

AWS has not said why, but part of the reason is structural. In Frankfurt, a model like Claude Sonnet 5.5 is not served in-Region at all: it runs through EU geographic cross-Region inference, which spreads requests over the EU Regions, or through global inference. ESC is one Region without cross-Region inference, so every model needs dedicated capacity in Brandenburg before it can appear. The catalogue will grow, but expect it to trail the commercial EU Regions at least until ESC has more than one Region. If your AI roadmap depends on frontier models, that alone may decide where those workloads live (see also When the model goes dark).

The practical rule: write your requirements at feature level, check them against Capabilities by Region and the per-service difference pages of the ESC User Guide, then test. The two sources do not always agree, which tells you how fast this is moving.

The ecosystem is not there yet

ESC has its own AWS Marketplace, with more than 165 solutions by May 2026, but it is a separate catalogue: only SaaS and AMI products are supported, and a SaaS listing must be deployable in ESC and keep its data there. Some vendors have shipped: AWS reports SAP Cloud ERP Private as generally available on ESC, and Sumo Logic launched its security operations platform there in June. Many others have announced intent and not shipped. At the time of writing:

• Snowflake, announced in 2025 as preparing for ESC, does not list an ESC Region on its supported-regions page.
• MongoDB said in 2025 it was preparing for ESC. The Atlas request for eusc-de-east-1 is at “Future Consideration”, with MongoDB saying in February that it is actively evaluating support.
• Trend Micro has announced plans for Vision One on ESC, without a date.
• Datadog’s AWS integration API accepts the partitions aws, aws-cn and aws-us-gov, not aws-eusc.

That is not a criticism of those vendors: building a SaaS control plane in a new partition is real work, and waiting for demand is a reasonable decision. But this gap matters more than most service gaps, for a reason that is easy to miss: your toolchain is inside your sovereignty boundary. ESC keeps your content and your metadata in the EU under EU operations. An observability, CSPM, backup or ITSM tool that reads your ESC accounts from a control plane outside the EU exports exactly what ESC was built to keep in. Before you design an ESC landing zone, list every tool that holds an IAM role in your accounts and check where each one runs.

Commercial, ESC or hybrid: how to decide

Start with the requirement, not the product. Three questions do most of the work.

1. Who requires sovereignty, and what exactly do they require? “Customer content stays in the EU” is met by any EU Region. “Only EU-resident staff operate the systems”, “customer-created metadata stays in the EU” or “the service keeps running when cut off from non-EU infrastructure” are where ESC earns its place. “A European-owned operator” is met by neither ESC nor any other US-parented option.
2. Does the workload fit inside today’s ESC? Services, features, models and every SaaS tool in the path.
3. Which data actually needs it? Data classification, not the application map, should draw the line.

The answer usually lands in one of three shapes.

AWS’s architecture guidance adds a fourth pattern: ESC as a sovereign failover target, with production in a commercial Region and backups, a pilot light or a warm standby in ESC. If your concern is geopolitical or legal disruption rather than day-to-day operation, backups or a tested pilot light are often the cheapest way to buy that exit option.

Whatever the shape, one rule decides whether hybrid works: the boundary has to follow the data. If the data you put in ESC gets copied into commercial for a dashboard, a model prompt or a support ticket, you have paid for the partition and kept the risk.

Hybrid means two clouds, and you pay for both

A hybrid estate is not one AWS environment with an extra Region. It is two environments that share an API style. AWS’s ESC landing zone guidance is explicit: a role in aws cannot assume a role in aws-eusc (or the reverse), and VPC peering, Transit Gateway, AWS RAM, S3 replication, ECR replication, AMI copy and VPC endpoints do not cross the boundary. In practice that means:

• Two organisations, two landing zones, each with its own management account, guardrails, logging and security services.
• Identity configured twice: a second IAM Identity Center instance federated to the same corporate identity provider, and CI/CD with one OIDC trust per partition, or a deployment plane inside ESC.
• Networking through the front door: TLS over the internet, Site-to-Site VPN or a separate Direct Connect connection. A commercial Direct Connect gateway cannot be extended into ESC.
• A separate supply chain: images and AMIs rebuilt or re-signed inside ESC rather than replicated, and certificates per partition.
• Infrastructure as code that knows about partitions. Anything that hardcodes arn:aws: breaks. Cheap to fix early, expensive to fix late:

# CloudFormation: let the template resolve the partition
Resource: !Sub "arn:${AWS::Partition}:s3:::${BucketName}/*"

# Terraform: same idea
data "aws_partition" "current" {}

locals {
  bucket_arn = "arn:${data.aws_partition.current.partition}:s3:::${var.bucket_name}"
}

Service endpoints move to amazonaws.eu, but IAM service principals such as lambda.amazonaws.com stay on amazonaws.com in both partitions.

Then there is the bill, where three costs stack up.

The unit price. ESC list prices are set in euro, at a figure slightly below the dollar price in Frankfurt. Lambda in ESC costs €0.0000164477 per GB-second against a commercial list price of $0.0000166667 in Frankfurt: about 1.3% lower as a number, but in a stronger currency. In dollar terms the premium is the exchange rate times 0.987, minus one: around 8.5% at 1.10 dollars per euro, around 18% at 1.20. A tecRacer comparison at launch, against Frankfurt, found roughly 15% across S3, FSx, EC2, Lambda, RDS and DynamoDB. Compared with Ireland, EC2 comes out higher, because Ireland is cheaper than Frankfurt to begin with. The flip side: commercial Regions can invoice in euro, but their prices are set in dollars and converted at the exchange rate of the invoice date. ESC list prices are set in euro, so the bill carries no currency risk, which some finance teams will value.

The commitment model. No EC2 Reserved Instances; EC2 commitments go through Savings Plans. Check the commitment options per service before you port an RI-based FinOps practice.

The duplication. A second landing zone, a second security tooling stack, a second identity configuration, a support plan in the second partition, data between partitions billed as data transfer out on the sending side, and the people who run it all. For a small ESC footprint, this fixed overhead easily exceeds the premium on the workload itself.

Which means hybrid is not automatically the cheaper option. When the workloads that could stay in commercial are small, or tightly coupled to the ones that must move, running two clouds and the integration between them can cost more than putting everything in ESC and paying the premium on the lot. Run both numbers before you decide to split.

ESC costs a little more per unit and a lot more per boundary. Put as little behind the boundary as your requirement allows, unless the boundary itself costs more than the premium you are trying to avoid.

Where Cloudar fits

This is the part where we talk about ourselves, so read it with that in mind. Cloudar is an ESC launch partner, holds the AWS Digital Sovereignty Competency, delivers Partner-Led Support in the AWS European Sovereign Cloud, and has built landing zones in the aws-eusc partition. That work has taught us that the hard part is rarely the ESC technology. It is keeping the moving parts consistent, so we have organised our work around the four questions customers actually ask:

Start with the Sovereignty Assessment. It is the step most organisations skip, and it decides whether the other three are needed at all. For many organisations the answer will be to stay in the commercial Regions with the right controls enforced. That is a valid outcome, and we will tell you so.

Saturday 24 October: AWS pulls the cable

Claims about operational independence are easy to write and hard to verify, so AWS is going to test one in public. On Saturday 24 October 2026, for several hours, ESC will run without its connection to the AWS Global Network backbone, the private network that normally carries authorised operational data between AWS locations. A dedicated system that handles limited, controlled transfers of operational data will be fully disabled, and traffic moves to ESC’s own internet connectivity from European ISPs. The exercise was called for by ESC’s managing directors and is run by a team made up entirely of EU residents.

What to expect. AWS says service availability inside ESC, other Regions and Direct Connect are not affected, but there may be brief connectivity disruptions at the start and end while routes converge on other networks. If you run cross-partition integrations over the internet or VPN, keep an eye on your monitoring that weekend.

What it proves. AWS designed the exercise to be consistent with the Commission’s Cloud Sovereignty Framework and BSI’s C3A, and will share an update with regulators and customers afterwards. That is the kind of evidence those frameworks ask for, and more than a policy document can offer.

Which brings us back to where we started. ESC is not the answer to every sovereignty question, and it is not just a label either. It is a partition: real operational separation, real gaps, real costs. The question is not whether ESC is sovereign enough. It is which of your data needs a partition, and whether you are prepared to run two clouds to give it one.

Service, feature and model availability in this post was checked against AWS Capabilities by Region, the ESC User Guide and the Amazon Bedrock model documentation on 16 October 2026. ESC is changing quickly; check before you plan.

Gartner, Magic Quadrant for Strategic Cloud Platform Services, 1 September 2026. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organisation and should not be construed as statements of fact. GARTNER and MAGIC QUADRANT are registered trademarks of Gartner, Inc. and/or its affiliates.

ISG Provider Lens, Multi Public Cloud Services, Sovereign Cloud Infrastructure Services (EU), 2025. ISG does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings. ISG research publications consist of the opinions of ISG’s research organisation and should not be construed as statements of fact.

The post The AWS European Sovereign Cloud is a partition, not a setting. Here is how to decide whether you need it. appeared first on Cloudar.

If you have never heard of the VDDK, that is normal. It is a C library, not a product. It is also the component that lets software outside the hypervisor open a VMDK and read blocks out of it. Combined with Changed Block Tracking, it is what makes agentless migration and agentless backup possible at all. Take it away and a large part of the tooling that reads VMware disks from the outside stops being installable.

Plenty of platforms are affected. This post is only about one of them: AWS. If you are running vSphere and you have been telling yourself that the migration project can wait another budget cycle, this is the part where the calculation changes.

What actually broke on the AWS side

AWS Transform MGN, still widely known as AWS Application Migration Service, offers two replication methods for VMware sources.

Agentless snapshot-based replication runs through the MGN vCenter Client, a software bundle you install on a dedicated Linux VM inside your vSphere environment. It discovers your VMs through vCenter, then for each replication run it takes a VMware snapshot, uses CBT to work out which blocks changed, and reads those blocks with the VDDK before shipping them to AWS.

The VDDK is not bundled with the client. It cannot be, because the licence forbids redistribution. You supply it yourself. The AWS documentation is explicit about the version:

You must log in to your Broadcomm account and download VDDK 7.0.3.3 to the VM on which the MGN vCenter Client is installed. VDDK 7.0.3.3 must be used, regardless of the vCenter version used.

The installer takes it as a parameter, --vddk-path, pointing at the tarball on local disk.

That is the whole problem in one line. VDDK 7.0.3.3 is a 7.0 line release. The 7.x downloads were already unreliable before August, and now the entire download tree is gone. As of writing, that documentation page is unchanged and still tells you to go and fetch a file that cannot be fetched. AWS has published nothing about the change: no bulletin, no what’s-new post, no documentation update. So a customer starting a greenfield agentless deployment today follows the runbook, hits a dead link at step four, and has no official guidance telling them why.

Agent-based replication installs the AWS Replication Agent inside the guest OS. It reads blocks through its own kernel-level driver, talks directly to a replication server in your staging subnet, and never touches the VDDK, vCenter or VMware snapshots. Nothing about it changed on 25 August.

Two things follow from that, and they matter more than the headlines suggest.

First, the practical damage on AWS is narrower than on platforms whose only migration path was VDDK-based. AWS’s own FAQ has said for years that agent-based replication is the default recommendation for all use cases, with agentless reserved for the exceptions: security policy that forbids installing agents, or an OS the agent does not support. The supported operating system lists for the two methods are in fact identical, so in practice the exception is almost always the policy one. Broadcom removed AWS’s fallback, not its main road.

Second, “narrower” is not “none”. If you are the organisation whose change board does not allow third-party agents in the guest, or you inherited an estate where nobody has working credentials for a few hundred VMs, agentless was the answer and it is now gone for new installs.

One nuance worth stating plainly because we have not seen AWS confirm it: an existing vCenter Client installation already has the VDDK on disk, and the client auto-updates itself independently of the VDDK. Running replications are not expected to break. But rebuild that VM, move to a new AWS account, or add a second vCenter environment, and you are back at the 404. Treat every existing vCenter Client as a single point of failure you can no longer reprovision.

Four routes to AWS that do not need a download from Broadcom

1. Agent-based MGN, deployed at scale with the MGN connector

The reflex objection to agent-based migration is that somebody has to log into every VM. That was true a few years ago. It is not the current state of the tooling, and this is the single most useful thing to know if the VDDK removal is what pushed you into a decision.

The MGN connector is a Linux VM in your source environment, registered to AWS through a Systems Manager hybrid activation, that pushes the replication agent out to your source servers for you. You give it network reachability to the sources and a set of source credentials in AWS Secrets Manager, and it handles prerequisite validation, installation on both Windows and Linux, and post-install verification. Installation of the connector itself takes a couple of minutes and is a one-time task per source environment. AWS Transform will now generate the IAM roles, the hybrid activation code and the ready-to-run install command for you through its console flow, or hand you a CloudFormation template if you would rather own the IAM yourself.

For the servers the connector cannot reach, the manual install is a single command. On Linux:

wget -O ./aws-replication-installer-init \
  https://aws-application-migration-service-eu-west-1.s3.eu-west-1.amazonaws.com/latest/linux/aws-replication-installer-init

chmod +x aws-replication-installer-init

sudo ./aws-replication-installer-init \
  --region eu-west-1 \
  --user-provided-id vm-prod-app-014 \
  --no-replication \
  --no-prompt

Two flags earn their keep on a large migration. --user-provided-id sets the identifier MGN uses to recognise the server consistently, which is also what AWS Transform keys on when it maps your inventory into wave plans, so set it at install time rather than trying to reconcile later. --no-replication installs without starting the data flow, which lets you pre-stage agents across hundreds of servers and then switch on replication wave by wave instead of saturating your uplink and your MGN quotas on day one.

Validate the installer hash before you push it estate-wide. It is published per region alongside the binary.

What you need in place, and what tends to be the actual project work rather than the agent install:

• Outbound TCP 443 from every source server to the regional MGN endpoint, and TCP 1500 to the replication servers in your staging subnet. PrivateLink over Direct Connect or Site-to-Site VPN if the data must not cross the internet.
• A staging area subnet with routing to S3, since the replication server fetches its own components from there. This is the single most common cause of a replication that installs cleanly and then stalls.
• Root or administrator on each guest. The Linux installer creates an aws-replication user and adds it to sudoers, and fails if it cannot.
• Kernel headers on the Linux guests matching the running kernel.

The compensation for the extra guest-level work is real: continuous block-level replication rather than periodic snapshot shipping, which means a cutover window measured in minutes instead of waiting for a final snapshot to complete, and no VMware snapshots sitting on your production datastores slowing disk performance during the whole migration.

2. The cached tarball, with a caveat you should not skip

If someone in your organisation still has VMware-vix-disklib-7.0.3.3-*.tar.gz on a build host, a backup appliance or an old runbook attachment, it will still work. The installer only wants a path. Checksums for the 7.0 and 8.0 lines are recoverable from cached copies of the portal, so verify what you hold rather than trusting the filename.

The caveat: Broadcom’s own wording to customers is that the VDDK is no longer available for use or download. Not just download. We are not lawyers and this is not legal advice, but if your plan depends on a cached copy of a proprietary library whose vendor has just told customers it is not available for use, that is a question for your legal team before it becomes a question for your auditor. Ask it now, not the week before cutover.

3. Offline export through VM Import/Export

This is the path that needs no VDDK and no agent at all. Export the VM from vSphere as an OVA or a stream-optimised VMDK using vCenter’s own export function or OVF Tool, upload it to S3, and import it as an AMI:

aws ec2 import-image \
  --disk-containers Format=ova,UserBucket="{S3Bucket=my-import-bucket,S3Key=vms/app014.ova}"

You need an IAM service role named vmimport with a trust policy for vmie.amazonaws.com and permissions on the source bucket and on EC2. Supported input formats are OVA, VMDK, VHD, VHDX and raw.

Be honest with yourself about what this path costs before you make it the plan for more than a handful of servers:

• No CBT, so every attempt is a full copy. A failed import is a full re-copy.
• Staging storage for the exports, on-premises and in S3, sized for your largest VMs.
• Real downtime, because you export a powered-off or quiesced VM and there is no delta sync afterwards.
• Genuine limitations. VMs using Raw Device Mapping are not supported, only VMDK disk images. Encrypted volumes are not supported. Dual-boot configurations are not supported. Neither is the VMware SEsparse delta-file format, nor arm64. Machines that got where they are through a physical-to-virtual conversion are not supported. There is a defined supported-OS matrix and older or unusual guests fall outside it.

VM Import/Export is a good tool for a controlled set of appliances and stragglers. It is not a replacement for a replication-based migration of a few hundred workloads, and anyone who tells you otherwise has not tried it at that scale.

4. Stop moving the VM

The uncomfortable observation in a lot of migration assessments is that a meaningful slice of the estate should not be rehosted at all. A file server becomes Amazon FSx or S3 with DataSync doing the copy. A SQL Server or Oracle instance becomes an RDS instance with AWS DMS handling the load and change data capture. An application server that was built by hand in 2016 gets rebuilt from a pipeline, which is faster than debugging why its agent will not install.

None of those paths care about the VDDK, because none of them read the virtual disk. If your agentless plan just died, this is a good moment to re-run the assessment and ask which VMs you were only lifting because lifting looked cheap.

What to do in the next two weeks

1. Inventory what you hold. Search your build hosts, backup appliances and documentation attachments for VDDK tarballs. Checksum them. Treat them as assets with an expiry date, and get the licensing question in front of legal.
2. Check whether you actually have an existing vCenter Client. If yes, freeze it. No rebuilds, no account moves, no second environment, until you have a tested alternative.
3. Test the agent on your hardest 5 percent. Not your easy Linux fleet. The 2012 R2 box nobody logs into, the appliance with a vendor-locked OS, the VM whose kernel headers are missing. Those are the servers that will decide whether agent-based works for you, and you want to know in September, not in March.
4. Exercise the OVA path once, deliberately. Export one real VM, import it, boot it, log in. A fallback you have never run is not a fallback.
5. Put the date on the wall. General support for vSphere 8 ends on 11 October 2027, with technical guidance running to 2029. That is your outer bound, and it is closer than it looks once you subtract assessment, network design, wave planning and testing.

The pattern is the point

Broadcom is reportedly bringing back a refreshed vSphere Standard aimed at smaller estates, discussed in interviews at VMware Explore rather than announced from a stage, with details promised in October. Read alongside the VDDK removal, the two moves are hard to reconcile as a coherent retention strategy. One says come back, the smaller edition is being looked after. The other quietly removes a dependency that the entire third-party exit ecosystem was built on, with no notice and no replacement.

Whatever the intent, the operational conclusion is the same. The technical routes off vSphere are being narrowed, not widened, and they are being narrowed without warning. The VDDK went away on a Tuesday. Nothing says the next dependency will give you more notice.

If you are going to leave, the cost of leaving is lowest while you still have working tooling, a supported hypervisor version, and time to test. That window is not open indefinitely.

Where Cloudar fits

We have done this often enough to know which parts of a VMware to AWS migration are the hard parts, and the disk copy is rarely one of them. The hard parts are the dependency mapping nobody documented, the network translation, the wave sequencing, the servers that resist every automated approach, and the operational handover afterwards. We work through those with AWS Transform and MGN where they fit, and around them where they do not.

If the VDDK removal has just invalidated your migration plan, or if it has made you realise you did not have one, our AWS migration practice is the place to start. We are an AWS Premier Partner and Migration Competency partner based in Belgium, and we would rather assess your estate honestly than sell you a lift-and-shift you will regret.

For the strategic side of the same question, why the timing of your VMware exit matters as much as the mechanics, see Leaving VMware is a question of timing.

All AWS documentation references were verified on 8 September 2026. AWS had published no guidance on the VDDK removal at that point, and the MGN vCenter Client requirements page still instructed customers to download VDDK 7.0.3.3 from Broadcom.

The post Broadcom just removed the VDDK. Here is how you still get from VMware to AWS. appeared first on Cloudar.

The problem? For most organisations, it delivered exactly what you’d expect when you relocate an existing environment to a new address: the same bottlenecks, the same inefficiencies, the same complexity. Just with a monthly invoice attached. Fast forward to today.

The invoice as a symptom

The appeal of lift & shift is understandable. No fundamental architecture changes. Limited impact on running applications. Minimal business disruption. A fast time-to-cloud that was easy to sell internally. For a specific type of workload – end-of-life systems or legacy environments you’rewinding down in the short term – it’s still a legitimate tactic today. Especially if it lets you shut down an entire data centre. But as a migration strategy, it’s postponement, not a real solution.

On-prem systems were designed for a world of fixed capacity. Servers run continuously whether the load justifies it or not. Redundancy lives in hardware, not in design. The economic logic of cloud is precisely the opposite: you pay for what you use and scale to what you need. That partnership between IT and Finance has a name: FinOps. The discipline that ties cloud spending to actual business value. Bringing an on-prem architecture to the cloud unchanged means importing the inefficiencies of the old model into an environment that was never built for them. And paying for that continuously.

In practice, this means: virtual machines running 24/7 when the workload doesn’t require it (idle resources), storage not tiered by access frequency, and applications moving data unnecessarily between regions or availability zones, each generating data transfer costs. Without FinOpsdiscipline, no one in the organisation has a clear picture of what the cloud actually costs per application, per team, or per business unit.

The result is an invoice that runs higher than the data centre you left, with less flexibility than you expected. Not because cloud is too expensive. But because you never made the shift from ‘infrastructure in the cloud’ to ‘cloud-native architecture with cost accountability’.On-prem doesn’t fix that. It just moves the problem and adds five-to-seven-year hardware depreciation cycles to the books. The costs become slightly less visible, but you’ll feel them regardless. Especially as hardware prices keep climbing due to supply constraints.

KarmAI

For years, the architectural debt of lift & shift was a slow-burning problem. Systems ran, bills were higher than expected, but business carried on. AI has changed that. Not gradually, but abruptly.

AI workloads require infrastructure that is fundamentally different from what classic enterprise applications need. Data must be accessible easily and at scale, not locked in legacy storage structures that were once migrated one-to-one. Integration with managed services like Amazon Bedrockworks better when your environment is built on cloud-native standards. AI workloads also have a volatile cost profile: a training job or inference spike can consume more in hours than a classic application does in a month. Without FinOps mechanisms to monitor, cap, and allocate those spikes, you lose all control over your AI budget.

But that’s only the short-term pain. Because organisations that did lift & shift also migrated their data mess along with everything else. No proper classification, no clear ownership, no consistent structure. For classic applications, that was annoying but manageable. For AI, it’s fatal. You can’tfine-tune a language model on unstructured data scattered across poorly managed cloud systems. You can’t build a RAG architecture if you don’t know where your data lives or what its quality is.

Organisations that are serious about AI are being forced to revisit their cloud architecture anyway. What once looked like a technical problem has become a strategic delay… right at the moment AI is starting to make the competitive difference. And organisations that now decide to return to on-prem entirely will only widen that gap.

On-prem is capitulation

Organisations considering cloud repatriation today are asking themselves the wrong question. Because the discussion isn’t really about cloud versus on-prem. It’s about focus and resource allocation: what does your organisation actually need, and are you capable of managing it well?

Is it five to midnight for some organisations? Yes. But for those who commit to a cloud-native approach, it will become clear that cloud was the right choice all along. And that anyone thinking about repatriation today will find there’s still a great deal of cloud potential left untapped.

If the cloud hasn’t delivered on its original promises, it’s usually because the architecture never became truly cloud-native. Data that was never properly structured. FinOps discipline that was never built in: no centralised cost reporting, no shared accountability between IT and business, no culture where teams think about the cost of what they build and run. And a mindset that carried on-prem patterns into an environment where they don’t belong.

You don’t solve those problems by switching environments. You solve them by acknowledging them and addressing them. And that can absolutely be done in the cloud… if you’re willing to do it differently this time.

The post Back to on-prem? appeared first on Cloudar.

In the AWS ecosystem, MSPs take vastly different approaches when building customer-specific landing zones and cloud management platforms. Some approaches preserve your freedom and flexibility. Others quietly build the walls of a gilded cage.

The Two Paths: Open Standards vs. Proprietary Platforms

The Open Approach: AWS Landing Zone Accelerator (LZA)

AWS Landing Zone Accelerator represents the gold standard for customer independence. As an open-source solution built on AWS CDK and CloudFormation, we deploy LZA to provide several critical advantages:

• Complete transparency: All infrastructure is defined as code that you can read, understand, and modify
• No licensing fees: Open-source with no proprietary components
• AWS-maintained: Continuously updated by AWS to keep up to date with new services and features
• Industry standard configuration: With multiple documented sample configurations, you do not need to start from scratch.
• Full customer ownership: Deployed directly into your environment with complete access to the Infrastructure as Code
• Exit-ready from day one: If you ever want to manage it yourself or switch MSPs, you own your landing zone configuration

At Cloudar, we’ve built an entire landing zone practice around LZA precisely because we believe customers should never feel trapped. Your AWS foundation should be an asset you own, not a chain that binds you to any single provider.

The Proprietary Approach: Custom Orchestration Platforms

In contrast, many MSPs have developed proprietary cloud management platforms that create significant lock-in. These are often marketed as “revolutionary” or “next-generation” platforms that promise to make cloud management easier with “just a few simple clicks” or web-based portals that abstract away complexity.

The convenience is real. The long-term cost is hidden.

Here’s what proprietary platforms typically involve:

• Black box deployment: Resources are created through proprietary tooling that abstracts away the underlying infrastructure
• Dependency on custom APIs: Your operations become dependent on the MSP’s platform rather than native AWS tools
• Limited portability: Moving to another MSP or bringing management in-house requires re-platforming
• Knowledge gap: Your team never develops deep expertise in AWS native tools because they’re shielded by the abstraction layer
• Commercial leverage: The MSP knows that switching costs are high, affecting pricing negotiations and service quality over time

The Lock-in Mechanisms You Need to Watch For

1. Custom Landing Zones Without Source Code Access

Some MSPs deploy your resources using “their” landing zone—a pre-configured multi-account setup built with proprietary Infrastructure as Code that remains their intellectual property. When you want to leave, you inherit an AWS environment you don’t fully understand, configured by tools you can’t access.

The Cloudar difference: We can deploy LZA directly into your AWS accounts. Every CloudFormation stack, every configuration file, every security baseline—you have access to it all.

2. Web-Based Orchestrators That Become Operational Chokepoints

Fancy web portals that let you “deploy with one click” sound appealing. Until you realize that every operational change must flow through the MSP’s platform. Want to modify a VPC? You’re dependent on their UI. Need to adjust security groups? Better hope their platform supports your use case.

These orchestrators create operational lock-in: You can’t effectively operate your AWS environment without the MSP’s tooling. You’ve traded AWS complexity for MSP dependency.

3. “Simplified” Interfaces That Hide AWS Reality

Abstraction layers that promise to “make AWS easy” can create a dangerous gap between what you think you’re deploying and what’s actually running in your account. When problems arise—and they always do—you discover that your team doesn’t understand the actual AWS infrastructure because they’ve only interacted with it through the MSP’s simplified interface.

The Real-World Impact of Lock-in

Scenario 1: The Price Increase

Your MSP announces a 30% price increase. With an LZA you own and open standards, you have options: negotiate from a position of strength, bring management in-house, or transition to another MSP in months. With a proprietary platform, you’re looking at risky and arduous re-platforming work—and your MSP knows it.

Scenario 2: The Service Quality Decline

Your MSP gets acquired. The new parent company shifts focus, key engineers leave, and service quality drops. With an open approach, you can transition smoothly. With lock-in, you’re stuck enduring declining service while planning an expensive migration.

Scenario 3: The Strategic Pivot

Your company wants to build internal cloud expertise and eventually self-manage. With LZA, your team can learn standard AWS tools and practices from day one. When you’re ready to transition, you already have the skills and the code. With proprietary platforms, your team has learned the MSP’s tools, not AWS—setting your in-house capability building back by years.

Scenario 4: The Platform Limitation

Your business needs evolve, and you need to implement a complex AWS architecture that isn’t supported by your MSP’s platform. You’re now in the worst position: paying for a platform that constrains you, unable to use native AWS capabilities, and facing the choice between living with limitations or undertaking an expensive re-platforming project.

How to Evaluate Your Current or Prospective MSP

Ask these critical questions:

1. “What landing zone solution do you use?”
   • Red flag: “Our proprietary solution” or vague answers. Subscription based landing zones (yes they exist!).
   • Green flag: “your own AWS Landing Zone Accelerator” or “a per-customer AWS Control Tower with Customizations for Control Tower”
2. “What happens to our infrastructure if we terminate the contract?”
   • Red flag: Vague answers about “transition planning” or “it depends”
   • Green flag: “You keep everything—we’ll help with knowledge transfer, and you’ll have all the code and documentation”
3. “Will our team learn AWS-native tools, or primarily your platform?”
   • Red flag: “Our platform abstracts AWS complexity away”
   • Green flag: “We teach AWS best practices and native tools”

The Cloudar Philosophy: Your Cloud, Your Terms

Here’s what that means in practice:

Full LZA Implementation

Every customer gets its own AWS Landing Zone Accelerator, deployed directly into their accounts with complete source code access to the LZA configuration. Deployments happen in your account, giving you end-to-end visibility on your Landing Zone.

AWS-Native Tooling

We use CloudFormation, CDK, AWS Config, Systems Manager—tools that work with or without us. If you hire another AWS expert or build an in-house team, they’ll recognize everything immediately.

Comprehensive Documentation

You can read about every configuration option today, in the documentation published by AWS. So while we pride ourselves in sharing our knowledge, you are not dependant on us to explain what is going on

Additionally, we write customer-specific documentation  in our Confluence – from architecture decisions to operational procedures.. If you decide to leave, we can provide you with an export of that information.

Open Book Operations

You have full Read Only access to your AWS environment —we’re partners, not gatekeepers. Want to check our work? Go ahead.

Standard AWS Best Practices

We follow AWS Well-Architected Framework principles and industry-standard patterns. No “special sauce” that only we understand.

We succeed by giving you excellent service so you want to stay, not by making it painful to leave.

The Economics of Freedom

Some argue that proprietary platforms are necessary to provide better service or lower costs. We disagree.

Lower costs come from:

• Automation that scales across customers (which we use)
• Deep AWS expertise (which we have)
• Efficient processes (which we’ve refined over years)

Not from locking customers into proprietary platforms.

Better service comes from:

• Highly skilled engineers (which we continuously train)
• Customer focus (which our retention rate proves)

Not from proprietary abstraction layers.

We’ve proven that you can deliver excellent MSP services at competitive prices while keeping customers completely free. In fact, we believe customer freedom makes us better—we can’t coast on lock-in, so we must continuously earn our customers’ business.

Making the Right Choice

Before signing with any MSP, ask yourself:

• Do I understand what I’m getting into? Can you clearly explain how your infrastructure will be deployed and managed?
• What’s my exit strategy if things don’t work out? Is it measured in weeks, months, or years?
• Am I choosing this approach for the right reasons? Is convenience masking a lack of control?
• Do I retain full ownership? Of code, configurations, documentation, and knowledge?

Red Flags in MSP Sales Processes

Be wary if you encounter:

• Heavy emphasis on “simplicity” with little discussion of the underlying AWS architecture
• Vague answers about exit strategies and transition processes
• Marketing focused on proprietary platforms as the primary differentiator
• Contracts that grant the MSP exclusive rights to infrastructure code
• Lack of clarity about what you actually own vs. what you’re licensing

Conclusion: Freedom as a Feature

In the rush to cloud transformation, it’s easy to prioritize speed and convenience. And yes, a well-designed proprietary platform can deploy faster than custom LZA implementation—at least initially.

But cloud strategy isn’t measured in weeks. It’s measured in years and decades. The question isn’t “who can get me to cloud fastest?” It’s “who can help me build sustainable cloud capabilities that serve my business long-term?”

The MSP industry has a pattern: some providers build their business model around customer stickiness achieved through proprietary tooling. They create beautiful interfaces and slick demos that abstract away AWS complexity. Then, months or years later, customers realize they’ve traded AWS vendor lock-in for MSP vendor lock-in—often worse, because at least AWS is standardized.

At Cloudar, we reject this model fundamentally. We believe that customer freedom isn’t a bug to work around—it’s a feature to build for. We’re proud to be an AWS Premier MSP Partner that wins business through excellence, not lock-in.

Your cloud infrastructure is too important to be held hostage by convenient abstractions. You deserve an MSP that treats you as a partner who will grow and evolve, not a captive customer who might someday try to escape.

Choose partners who believe you should always have the keys to your own kingdom. Choose partners who succeed by being valuable, not by being necessary.

Choose freedom.

The post The Hidden Cost of Convenience appeared first on Cloudar.

Interestingly, Roles Anywhere does not completely solve the problem of not needing to create long-lived credentials, instead it moves (and reduces) the problem from credential management (rotation) to certificate management (distribution).

Setting up a Certificate Authority (CA) that can create certificates can be done with AWS Private Certificate Authority, and there are some options out there to manage distribution (usually as part of a bigger Device Management solution), but wouldn’t it be nice if the government of Belgium would do that for us?

The Belgian eID

In Belgium every citizens gets an identity card. The government makes sure that these get distributed and renewed, and since 2005 they contain a chip that holds (among other things) a certificate that can be used for digital authentication. If we can tie this to Roles Anywhere, we don’t have to worry about certificate management ourselves.

This eID is the size of a bankcard and the chip can be interacted with using off the shelf smart card readers. Most information can be read without any additional authentication – if you have the correct middleware, but to sign something you need to enter a PIN code.

The different certificates

Every eID holds an “authentication certificate” that is tied to the owner of the eID and has been signed by the “Citizens CA”.  The Citizens CA’s certificate is in turn signed by the Belgium Root CA . The Root certificate is self-signed. If we trust the Belgium Root CA, we will be able to validate any authentication certificate. More information about this setup can be found at https://repository.eidpki.belgium.be/#/home

The authentication certificate has two pieces of information that are going to be useful to us:

• The Issuer, showing us that the certificate is indeed issued by the Citizen CA.
• The Subject, containing the name and National Number (a number that uniquely points to one person) of the owner.

AWS IAM Roles Anywhere

Roles Anywhere allows you to use a certificate and its private key to get temporary credentials. You can think of it as an AssumeRole call, but instead of starting from an Access Key, we start from a certificate.

There are a few components we need:

• Configuration in our AWS Account, to indicate which CA we trust, and which roles can be assumed by different Issuer and/or Subjects. This also includes the creation of IAM Roles.
• Software on our computer to interact with the Card Reader so we can prove we have access to the eID and know its PIN.

For our use case we are going to trust the Belgium Root CA, and create a Role that can be assumed with any certificate issued by the Citizen CA. We will than limit the permissions of that IAM Role, so it can only write to a unique prefix in S3.

We can do that because every component of the Subject in our certificate gets mapped to a principal tag on the role session. E.g. if the subject is “C=BE, SN=Doe, GN=John, serialNumber=70010112345, CN=John Doe (Authentication)”, we have the following principal tags, and we can use principal tags in any policy.

• x509Subject/C: BE
• x509Subject/SN: Doe
• x509Subject/GN: John
• x509Subject/serialNumber: 70010112345
• x509Subject/CN: John Doe (Authentication)

Putting it all together

In our AWS Account

We will create the following resources in our AWS Account:

• An S3 Bucket
• A trust anchor, configured to trust the Belgium Root CA certificate that is publicly available via https://crt.eidpki.belgium.be/eid/brca6.crt.  This way we can use any certificate created by the Root CA of any of its subordinate CAs (like the Citizen CA)
• An IAM Role, where we configure the trust policy to only allow usage of the Role if:
  • The principal is the Roles Anywhere service (rolesanywhere.amazonaws.com)
  • The Issuer is Citizen CA (aws:PrincipalTag/x509Issuer/CN must equal “Citizen CA”)
  • The source is our trust anchor (aws:SourceArn equals the ARN of the Root CA trust anchor)
• An IAM Policy on this role that allows PutObject acces to all objects that start with the unique National Number (aws:PrincipalTag/x509Subject/serialNumber)
• a (Roles Anywhere) profile that ties the trust anchor to the Role. We could configure more settings here, like a mapping of fields from the certificate or a session policy, but don’t need to.

Of course we do not have to do this by hand, we create a CloudFormation template for this. It can be found in our sample repository: https://github.com/WeAreCloudar/cloudformation-samples/tree/main/templates/roles-anywhere-eid

On our machine

On our machine, we will need – besides the eID and its PIN (at least on macOS, where I tested this):

• a SmartCard Reader
• the Roles Anywhere Credential Helper (downloadable from the AWS Website)
• The AWS CLI (or any other tool we want to use with our AWS Credentials)

We also need a few pieces of information:

• The ARNs of the trust anchor, profile and role we created
• Our own National Number and the name of the bucket (to test our policy)

Ideally we’d be able to configure the credential helper to read any eID we insert, and this is possible if you install the Belgian eID Middleware using the following command.

aws_signing_helper credential-process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--certificate "pkcs11:model=Belgium%20eID;object=Authentication" \
--pkcs11-lib '/Library/Belgium Identity Card/Pkcs11/beid-pkcs11.bundle/Contents/MacOS/libbeidpkcs11.dylib

However because of an issue with the helper, this only works on macOS if you build the helper yourself from source. Since we do not want to do that, we can plug in our eID and find and identifier using “aws_signing_helper read-certificate-data”. You should get something like

Matching identities
1) bf013dd57aa34f8f9d4e22ab89dd8eb2 "SERIALNUMBER=70010112345,CN=John Doe (Authentication),C=BE,..."
2) 786f7572198a4a8a82ee7ef3f205d7be "SERIALNUMBER=70010112345,CN=John Doe (Signature),C=BE,..."


Copying everything between quotes, allows us to create a new file, selector.json that looks like this:


[
{
"Key": "x509Subject",
"Value": "SERIALNUMBER=70010112345,CN=John Doe (Authentication),C=BE,..."
}
]

and use that to select our specific certificate:

aws_signing_helper credential-process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--cert-selector file://selector.json

Our smart card reader will now prompt us for a PIN. If we enter the right one, we get credentials as output:

{"Version":1,"AccessKeyId":"ASIA...","SecretAccessKey":"COA...","SessionToken":"IQoJ...","Expiration":"2025-03-28T14:16:07Z"}


We can put this command in our ~/.aws/config file, so the AWS CLI will execute it for us, but for testing purposes it’s slightly easier to use the “serve” option.  The AWS Documentation has a longer explanation about how you can use the credential-process: https://docs.aws.amazon.com/rolesanywhere/latest/userguide/credential-helper.html#credential-helper-examples

Testing with s3

In one terminal we can run a credential / metadata server that can be used by the AWS CLI:

~$ aws_signing_helper serve -process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--cert-selector file://selector.json


This gives us
2025/03/28 14:28:12 Local server started on port: 9911
2025/03/28 14:28:12 Make it available to the sdk by running:
2025/03/28 14:28:12 export AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:9911/

We can than use a second terminal to run:

AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:9911/ aws s3 cp example_file s3://$bucket/serialNumber/70010112345/example_file

and get our file in s3:

upload: example_file to s3://.../serialNumber/70010112345/example_file

Conclusion

This experiment shows that being able to use an external device withIAM Roles, is a very powerful feature, because it can remove a lot of tasks related to certificate and credential management. Being able to do this with something I had in my backpocket made it extra cool.

However, the Belgian eID is probably not the best way to go about this in a real scenario:

• I just gave everyone living in Belgium access to my S3 Bucket. I could use a more scoped-down trust policy, but I would eventually run into limitations of trust-policy size.
• I also didn’t tackle Certificate Revocation Lists (CRL). Every time someone loses their wallet, I would need to invalidate certain certificates – this also runs into size limitations
• I already have a good way to give users access to AWS Accounts, using Identity Center or Cognito.
• The pkcs11 support on macOS seems limited, and I would have to install the middleware everywhere,

This does not mean that Roles Anywhere is not a good solution for this kind of cases:

• If you are already putting certificates on machines (e.g using an MDM solution), AWS IAM Roles Anywhere is a great fit
• Instead of relying on the Belgian Government you can buy an hardware key like a yubikey and use it as your certificate source. This is as secure as using the eID (only the machine with the key plugged in will be able to request credentials), and you would be limiting it to the exact serial number(s) in your trust policy anyway – making CRLs mostly moot.

The post Sign in with your eID: Using AWS IAM Roles Anywhere with a SmartCard Reader appeared first on Cloudar.

We were recently working on an AWS setup which involved a Network LoadBalancer (NLB) with a TCP listener and a requirement for sticky sessions. As we were seeing some strange behavior which we couldn’t immediately explain and which might be linked to the session stickiness we decided to make a small test setup.

The problem

Unlike an ALB where session stickiness is accomplished with cookies, the NLB uses a built-in 5-tuple hash table in order to maintain stickiness across backend servers. We access the NLB through its DNS name, which actually returns the IPs of the two NLB endpoints in a round-robin fashion with a TTL of 60 seconds.

We were looking for an answer on the following question: if our end-user would resolve the DNS and pick the IP of the first NLB endpoint to start the connection, the session will be routed towards one of the backend servers. But after 60 seconds the client could potentially re-issue the DNS query and start its connection with the other NLB endpoint. How will the stickiness and cross-zone loadbalancing behave? Will our end-user connection be routed to the initial server again, even if that means crossing AZ boundaries?

The situation

We started with a classical setup comprising of an NLB with an endpoint in each AZ and a Targetgroup having one instance as target in each AZ.

Scenario #1

• Cross-Zone loadbalancing: Disabled
• TargetGroup Stickiness: Disabled

How does it behave?

1. Client connects to the IP of the first NLB node: the connection is redirected to the server in AZ 1.
2. Client connects to the IP of the second NLB node: the connection is redirect to the server in AZ 2.

Since there is only one healthy target per AZ and cross-zone loadbalancing is not enabled, this situation results in ‘AZ-stickiness’: the traffic remains in the AZ in which it arrived. The setup relies on DNS to distribute client connections evenly across both NLB endpoints, but there is nothing to guarantee that a specific user connection is always directed to the same NLB endpoint, let alone to the same backend server.

Scenario #2

• Cross-Zone loadbalancing: Enabled
• TargetGroup Stickiness: Disabled

Allowing cross-zone loadbalancing and not requiring any stickness. This should give us complete randomness, shouldn’t it?

And so it does. We’re now completely randomized and hit every backend server, irrespective of our NLB endpoint ‘point of entry’. Works as expected.

Scenario #3

• Cross-Zone loadbalancing: Disabled
• TargetGroup Stickiness: Enabled

We’ve enabled stickiness on the targetgroup now, and disabled the cross-zone loadbalancing again. Let’s hope our client connection is now sticky to a specific backend server.

1. Client connects to the IP of the first NLB node: the connection is redirected to the server in AZ 1.
2. Client connects to the IP of the second NLB node: the connection is redirect to the server in AZ 2.

Ok wait, we’ve asked our TargetGroup to be sticky, but still our connection is balanced over both backend servers? What’s going on?

The fact that our NLB is not allowing cross-zone loadbalancing seems to prevent the connection from reaching the same backend every time. The connection enters via NLB endpoint 1 but stickiness has decided that the connection should go to server in AZ 2? Stickiness fails, the disabled cross-zone loadbalancing wins…

With only one healthy backend per AZ this behaves the same as not enabling stickiness at all. We’re pretty sure that with more than one backend per AZ the stickiness is maintained…within that AZ only. Interesting!

Scenario #4

• Cross-Zone loadbalancing: Enabled
• TargetGroup Stickiness: Enabled

Let’s solve this. We’ve enabled both cross-zone loadbalancing and targetgroup stickiness. We should hit the same backend server every time now.

And so it does. Only now we reach true stickiness and hit the same backend server every time, no matter how hard we try by entering via the loadbalancer node in the other AZ.

The conclusion

If you don’t allow cross-zone loadbalancing, then stickiness is only active within AZ boundaries. As DNS round-robin could direct a client to a different point of entry after the TTL has expired, strict stickiness is not guaranteed.

So if you really need stickiness to a specific backend target, you need to allow cross-zone loadbalancing (and live with the extra cost of inter-AZ traffic). Only now do the different loadbalancer nodes share the hash table of “client-to-target” stickiness.

Kinda logic, though…

PS: NLB idle timeout for TCP connections is 350 seconds. Once the timeout is reached or the session is terminated, the NLB will forget the stickiness and incoming packets will be considered as a new flow and could be loadbalanced to a new target.

The post Why AWS NLB stickiness is not always sticky appeared first on Cloudar.

The post 10 random things you probably didn’t know about Cloudar appeared first on Cloudar.

It’s a familiar scene: as you’re waking up or making your morning coffee, you hear the garbage truck roaring away outside when you suddenly realize your garbage bags are still rotting away in the garage. Do you run after the truck like a lunatic, even though it’s too late (of course I’ve never done this, but I’ve heard it’s a thing)? Or do you just let it go and let the garbage fester even longer? Either way, it’s a frustrating start to the day.

Luckily for me I live in the modern and very civilised province of Limburg of Belgium where the local government has made a website available with a downloadable calendar showing all dates the garbage truck comes along: so handy! They even provide the calendar in a number of digital formats, including iCal. I could just import this into my phone and set an alert on it, but where’s the fun in that? After some fiddling around or “reverse engineering” as they call it, I found out that the data for the calendar files was provided by an undocumented but publicly available API.

Being the sucker for coding up and fooling around with APIs that I am (and given my state of utter boredom while under full lockdown in Belgium) I decided to code up a project based on this API that would send me an SMS notification the evening before garbage collection. Because that’s my idea of fun. And that way I would be reminded to take out the trash on time and keep my garage clean.

Want to know this neat trick to save you from rubbish resentment? Read on!

So for this project I partly used the AWS SAM framework and set up a CodePipeline to deploy my code changes instantly on AWS (stay tuned for a future blogpost on this).

First let’s code up the AWS SAM template file containing the necessary resources for this project.

The resources I used are:

• Type: AWS::Events::Rule
  A Cloudwatch event rule to invoke a lambda each evening around 8.
• Type: AWS::Lambda::Permission
  To give the Cloudwatch event rule permissions to invoke the lambda.
• Type: AWS::Logs::LogGroup
  A log group for the lambda logs.
• Type: AWS::Serverless::Function
  The lambda that will run the show.
• Type: AWS::IAM::Role
  An IAM role for the lambda to have the necessary rights like accessing SNS for the SMS notification.

Next we code up the lambda that will call the API and send us an SMS.

The API provides event data for each month by just requesting the year and month in the url path like this ‘/2020-03’. So we can easily get the data for the current month and loop through its events, checking whether any are occurring the next morning. If so we receive an SMS notifying us (note that you can provide the number of your girlfriend here, but it’s not recommended as I have found out it isn’t that effective).

And that’s all folks!

I’m aware that this trick is no rocket science, and that this isn’t even a full guide to get to the same result (leave a comment if you want some more details), but I thought it would be fun to share how I resolved one of life’s little problems using AWS.

As I’m moving toward two years of professional experience using the platform, I’m still amazed at how easily you can code some services together and create a working, scalable and highly available project in a matter of minutes or hours.

Hope you liked my trashy tale – keep your garages clean and your heads in the cloud!

The post Let’s trash talk! appeared first on Cloudar.

We will use a custom resource written in Python that will be able to create ACM certificates with DNS validation. The custom resource will also automatically validate this certificate if the validation domain is managed by a Route53 hosted zone. We will also be able to specify an AWS region to create the certificate in, this region is independent of the Cloudformation stack region which for example makes it possible to deploy a certificate in region us-east-1 (to use with cloudfront) while deploying the stack in region eu-west-1. The resource will also provide the certificate arn as an output parameter so it can be used by other resources in the stack. Lastly when you delete the custom resource it will cleanup all validation records and the certificate itself.

Requirements:

• Python3
• Pip
• Bash
• Zip
• An S3 bucket to deploy the custom resource package on
• A hosted zone for the validation record

Implementation:

Let’s get started by downloading all the required code from our GitHub repository.

Step1: Uploading the custom resource package

In this step we are going to prepare the custom resource package and upload it to an S3 bucket.

First we go into the custom resource directory.
cd cloudar-acm-plus-custom-resource

Next we execute a script to install all required dependencies.
sh install_dependencies

Now we are ready to create the package.
sh pack_custom_resource

You will now find the zipfile ‘cloudar-acm-plus-custom-resource.zip’ in ‘cloudar-acm-plus-custom-resource/packed’, upload this zip file to your S3 bucket.

Step2: Creating a Cloudformation template

Now we can create a Cloudformation template in which we use this custom resource to create an ACM certificate.
You can use the template ‘cfn.yaml’ as an example.

First create a Lambda resource as following

Use the name of your bucket for the property ‘S3Bucket’ .

Next we create the custom resource.

We can set the following properties here:

• DomainName: (REQUIRED type:String) The domain name for the acm certificate.
• AdditionalDomains: (OPTIONAL type:List) Additional domains for the acm certificate
• ValidationDomain: (REQUIRED type:string) The domain name for the validation domain of the acm certificate
• HostedZoneId: (REQUIRED type:string) The hosted zone id for the validation domain of the acm certificate
• CertificateRegion: (REQUIRED type:string) The region to deploy the acm certificate in
• IdempotencyToken: (REQUIRED type:string pattern: \w+) The idempotency token for the create call of the acm certificate doc: https://docs.aws.amazon.com/acm/latest/APIReference/API_RequestCertificate.html#ACM-RequestCertificate-request-IdempotencyToken
• CertificateTags: (OPTIONAL type:list) The tags for the acm certificate

In order for the DNS record cleanup and delete certificate functionality to work when you delete the Cloudformation stack it is important to set the following output.

As you can see we can access the arn of the certifcate created by the custom resource with the GetAtt function on the resource.
!GetAtt CreateCertificateCustomResource.certificate_arn

Step3: Deploy the cloudformation

Finally the only thing left to do is deploy the Cloudformation template.
Once the deploy is started Cloudformation will create the Lambda containing the code from step1 and start a custom resource which will create the certificate and validation records. Once the status of the certificate becomes ‘ISSUED’ the custom resource will finish successfully and report the arn of the certificate back to Cloudformation. We can now further use this arn in other resources in the Cloudformation template.
When you delete the Cloudformation stack, the custom resource will cleanup the validation records in the hosted zone and delete the certificate.

CREATE_COMPLETE

The post Validate ACM certificates in Cloudformation appeared first on Cloudar.

Video isn’t the best reference material if you want to quickly find one of those tips, so I will add a quick summary of them in this blogpost.

• By specifying source_profile in your .aws/config file, you can tell the AWS CLI to use credentials from one profile to assume the role_arn you configure in another profile. This also works in a chain of profiles and can include asking for an MFA token.
• You don’t have to get credentials from the .aws/credentials file, you can tell the CLI to get them from environment variables (this also works if you do not specify anything), instance metadata, or the ECS container role.
• There is a --debug flag, which gives you a lot of output to see everything that’s happening behind the scenes.
• In most cases you do not want that full debug output. Enabling history and using aws history show will give you the most relevant parts in an easy to read format.
• The CLI has the annoying habit of resolving all urls you give it. You can disable this with aws configure set cli_follow_urlparam false.
• You can make S3 uploads and downloads go a lot faster by tweaking the max_concurrent_requests. Or make it not exceed a certain speed with max_bandwith. See here for all options.
• The cli supports defining your own aliases. Look in this awslabs repository for examples.
• Instead of relying on an external tool (that needs to be installed), you can get specific fields from the response by using --query. Combine this with --output text to strip json-specific syntax.
• You can use completion in different shells.
• There are waiters you can use directly from the command line, so you do not have to write your own check-sleep loops in bash.
• Other useful cli tools include cfn-lint, aws-encryption-sdk, aws-shell and aws-vault
• There is a version 2 coming.

The post $ aws help # things you might not know about the AWS CLI appeared first on Cloudar.