If you have never heard of the VDDK, that is normal. It is a C library, not a product. It is also the component that lets software outside the hypervisor open a VMDK and read blocks out of it. Combined with Changed Block Tracking, it is what makes agentless migration and agentless backup possible at all. Take it away and a large part of the tooling that reads VMware disks from the outside stops being installable.

Plenty of platforms are affected. This post is only about one of them: AWS. If you are running vSphere and you have been telling yourself that the migration project can wait another budget cycle, this is the part where the calculation changes.

What actually broke on the AWS side

AWS Transform MGN, still widely known as AWS Application Migration Service, offers two replication methods for VMware sources.

Agentless snapshot-based replication runs through the MGN vCenter Client, a software bundle you install on a dedicated Linux VM inside your vSphere environment. It discovers your VMs through vCenter, then for each replication run it takes a VMware snapshot, uses CBT to work out which blocks changed, and reads those blocks with the VDDK before shipping them to AWS.

The VDDK is not bundled with the client. It cannot be, because the licence forbids redistribution. You supply it yourself. The AWS documentation is explicit about the version:

You must log in to your Broadcomm account and download VDDK 7.0.3.3 to the VM on which the MGN vCenter Client is installed. VDDK 7.0.3.3 must be used, regardless of the vCenter version used.

The installer takes it as a parameter, --vddk-path, pointing at the tarball on local disk.

That is the whole problem in one line. VDDK 7.0.3.3 is a 7.0 line release. The 7.x downloads were already unreliable before August, and now the entire download tree is gone. As of writing, that documentation page is unchanged and still tells you to go and fetch a file that cannot be fetched. AWS has published nothing about the change: no bulletin, no what’s-new post, no documentation update. So a customer starting a greenfield agentless deployment today follows the runbook, hits a dead link at step four, and has no official guidance telling them why.

Agent-based replication installs the AWS Replication Agent inside the guest OS. It reads blocks through its own kernel-level driver, talks directly to a replication server in your staging subnet, and never touches the VDDK, vCenter or VMware snapshots. Nothing about it changed on 25 August.

Two things follow from that, and they matter more than the headlines suggest.

First, the practical damage on AWS is narrower than on platforms whose only migration path was VDDK-based. AWS’s own FAQ has said for years that agent-based replication is the default recommendation for all use cases, with agentless reserved for the exceptions: security policy that forbids installing agents, or an OS the agent does not support. The supported operating system lists for the two methods are in fact identical, so in practice the exception is almost always the policy one. Broadcom removed AWS’s fallback, not its main road.

Second, “narrower” is not “none”. If you are the organisation whose change board does not allow third-party agents in the guest, or you inherited an estate where nobody has working credentials for a few hundred VMs, agentless was the answer and it is now gone for new installs.

One nuance worth stating plainly because we have not seen AWS confirm it: an existing vCenter Client installation already has the VDDK on disk, and the client auto-updates itself independently of the VDDK. Running replications are not expected to break. But rebuild that VM, move to a new AWS account, or add a second vCenter environment, and you are back at the 404. Treat every existing vCenter Client as a single point of failure you can no longer reprovision.

Four routes to AWS that do not need a download from Broadcom

1. Agent-based MGN, deployed at scale with the MGN connector

The reflex objection to agent-based migration is that somebody has to log into every VM. That was true a few years ago. It is not the current state of the tooling, and this is the single most useful thing to know if the VDDK removal is what pushed you into a decision.

The MGN connector is a Linux VM in your source environment, registered to AWS through a Systems Manager hybrid activation, that pushes the replication agent out to your source servers for you. You give it network reachability to the sources and a set of source credentials in AWS Secrets Manager, and it handles prerequisite validation, installation on both Windows and Linux, and post-install verification. Installation of the connector itself takes a couple of minutes and is a one-time task per source environment. AWS Transform will now generate the IAM roles, the hybrid activation code and the ready-to-run install command for you through its console flow, or hand you a CloudFormation template if you would rather own the IAM yourself.

For the servers the connector cannot reach, the manual install is a single command. On Linux:

wget -O ./aws-replication-installer-init \
  https://aws-application-migration-service-eu-west-1.s3.eu-west-1.amazonaws.com/latest/linux/aws-replication-installer-init

chmod +x aws-replication-installer-init

sudo ./aws-replication-installer-init \
  --region eu-west-1 \
  --user-provided-id vm-prod-app-014 \
  --no-replication \
  --no-prompt

Two flags earn their keep on a large migration. --user-provided-id sets the identifier MGN uses to recognise the server consistently, which is also what AWS Transform keys on when it maps your inventory into wave plans, so set it at install time rather than trying to reconcile later. --no-replication installs without starting the data flow, which lets you pre-stage agents across hundreds of servers and then switch on replication wave by wave instead of saturating your uplink and your MGN quotas on day one.

Validate the installer hash before you push it estate-wide. It is published per region alongside the binary.

What you need in place, and what tends to be the actual project work rather than the agent install:

• Outbound TCP 443 from every source server to the regional MGN endpoint, and TCP 1500 to the replication servers in your staging subnet. PrivateLink over Direct Connect or Site-to-Site VPN if the data must not cross the internet.
• A staging area subnet with routing to S3, since the replication server fetches its own components from there. This is the single most common cause of a replication that installs cleanly and then stalls.
• Root or administrator on each guest. The Linux installer creates an aws-replication user and adds it to sudoers, and fails if it cannot.
• Kernel headers on the Linux guests matching the running kernel.

The compensation for the extra guest-level work is real: continuous block-level replication rather than periodic snapshot shipping, which means a cutover window measured in minutes instead of waiting for a final snapshot to complete, and no VMware snapshots sitting on your production datastores slowing disk performance during the whole migration.

2. The cached tarball, with a caveat you should not skip

If someone in your organisation still has VMware-vix-disklib-7.0.3.3-*.tar.gz on a build host, a backup appliance or an old runbook attachment, it will still work. The installer only wants a path. Checksums for the 7.0 and 8.0 lines are recoverable from cached copies of the portal, so verify what you hold rather than trusting the filename.

The caveat: Broadcom’s own wording to customers is that the VDDK is no longer available for use or download. Not just download. We are not lawyers and this is not legal advice, but if your plan depends on a cached copy of a proprietary library whose vendor has just told customers it is not available for use, that is a question for your legal team before it becomes a question for your auditor. Ask it now, not the week before cutover.

3. Offline export through VM Import/Export

This is the path that needs no VDDK and no agent at all. Export the VM from vSphere as an OVA or a stream-optimised VMDK using vCenter’s own export function or OVF Tool, upload it to S3, and import it as an AMI:

aws ec2 import-image \
  --disk-containers Format=ova,UserBucket="{S3Bucket=my-import-bucket,S3Key=vms/app014.ova}"

You need an IAM service role named vmimport with a trust policy for vmie.amazonaws.com and permissions on the source bucket and on EC2. Supported input formats are OVA, VMDK, VHD, VHDX and raw.

Be honest with yourself about what this path costs before you make it the plan for more than a handful of servers:

• No CBT, so every attempt is a full copy. A failed import is a full re-copy.
• Staging storage for the exports, on-premises and in S3, sized for your largest VMs.
• Real downtime, because you export a powered-off or quiesced VM and there is no delta sync afterwards.
• Genuine limitations. VMs using Raw Device Mapping are not supported, only VMDK disk images. Encrypted volumes are not supported. Dual-boot configurations are not supported. Neither is the VMware SEsparse delta-file format, nor arm64. Machines that got where they are through a physical-to-virtual conversion are not supported. There is a defined supported-OS matrix and older or unusual guests fall outside it.

VM Import/Export is a good tool for a controlled set of appliances and stragglers. It is not a replacement for a replication-based migration of a few hundred workloads, and anyone who tells you otherwise has not tried it at that scale.

4. Stop moving the VM

The uncomfortable observation in a lot of migration assessments is that a meaningful slice of the estate should not be rehosted at all. A file server becomes Amazon FSx or S3 with DataSync doing the copy. A SQL Server or Oracle instance becomes an RDS instance with AWS DMS handling the load and change data capture. An application server that was built by hand in 2016 gets rebuilt from a pipeline, which is faster than debugging why its agent will not install.

None of those paths care about the VDDK, because none of them read the virtual disk. If your agentless plan just died, this is a good moment to re-run the assessment and ask which VMs you were only lifting because lifting looked cheap.

What to do in the next two weeks

1. Inventory what you hold. Search your build hosts, backup appliances and documentation attachments for VDDK tarballs. Checksum them. Treat them as assets with an expiry date, and get the licensing question in front of legal.
2. Check whether you actually have an existing vCenter Client. If yes, freeze it. No rebuilds, no account moves, no second environment, until you have a tested alternative.
3. Test the agent on your hardest 5 percent. Not your easy Linux fleet. The 2012 R2 box nobody logs into, the appliance with a vendor-locked OS, the VM whose kernel headers are missing. Those are the servers that will decide whether agent-based works for you, and you want to know in September, not in March.
4. Exercise the OVA path once, deliberately. Export one real VM, import it, boot it, log in. A fallback you have never run is not a fallback.
5. Put the date on the wall. General support for vSphere 8 ends on 11 October 2027, with technical guidance running to 2029. That is your outer bound, and it is closer than it looks once you subtract assessment, network design, wave planning and testing.

The pattern is the point

Broadcom is reportedly bringing back a refreshed vSphere Standard aimed at smaller estates, discussed in interviews at VMware Explore rather than announced from a stage, with details promised in October. Read alongside the VDDK removal, the two moves are hard to reconcile as a coherent retention strategy. One says come back, the smaller edition is being looked after. The other quietly removes a dependency that the entire third-party exit ecosystem was built on, with no notice and no replacement.

Whatever the intent, the operational conclusion is the same. The technical routes off vSphere are being narrowed, not widened, and they are being narrowed without warning. The VDDK went away on a Tuesday. Nothing says the next dependency will give you more notice.

If you are going to leave, the cost of leaving is lowest while you still have working tooling, a supported hypervisor version, and time to test. That window is not open indefinitely.

Where Cloudar fits

We have done this often enough to know which parts of a VMware to AWS migration are the hard parts, and the disk copy is rarely one of them. The hard parts are the dependency mapping nobody documented, the network translation, the wave sequencing, the servers that resist every automated approach, and the operational handover afterwards. We work through those with AWS Transform and MGN where they fit, and around them where they do not.

If the VDDK removal has just invalidated your migration plan, or if it has made you realise you did not have one, our AWS migration practice is the place to start. We are an AWS Premier Partner and Migration Competency partner based in Belgium, and we would rather assess your estate honestly than sell you a lift-and-shift you will regret.

For the strategic side of the same question, why the timing of your VMware exit matters as much as the mechanics, see Leaving VMware is a question of timing.

All AWS documentation references were verified on 8 September 2026. AWS had published no guidance on the VDDK removal at that point, and the MGN vCenter Client requirements page still instructed customers to download VDDK 7.0.3.3 from Broadcom.

The post Broadcom just removed the VDDK. Here is how you still get from VMware to AWS. appeared first on Cloudar.

The problem? For most organisations, it delivered exactly what you’d expect when you relocate an existing environment to a new address: the same bottlenecks, the same inefficiencies, the same complexity. Just with a monthly invoice attached. Fast forward to today.

The invoice as a symptom

The appeal of lift & shift is understandable. No fundamental architecture changes. Limited impact on running applications. Minimal business disruption. A fast time-to-cloud that was easy to sell internally. For a specific type of workload – end-of-life systems or legacy environments you’rewinding down in the short term – it’s still a legitimate tactic today. Especially if it lets you shut down an entire data centre. But as a migration strategy, it’s postponement, not a real solution.

On-prem systems were designed for a world of fixed capacity. Servers run continuously whether the load justifies it or not. Redundancy lives in hardware, not in design. The economic logic of cloud is precisely the opposite: you pay for what you use and scale to what you need. That partnership between IT and Finance has a name: FinOps. The discipline that ties cloud spending to actual business value. Bringing an on-prem architecture to the cloud unchanged means importing the inefficiencies of the old model into an environment that was never built for them. And paying for that continuously.

In practice, this means: virtual machines running 24/7 when the workload doesn’t require it (idle resources), storage not tiered by access frequency, and applications moving data unnecessarily between regions or availability zones, each generating data transfer costs. Without FinOpsdiscipline, no one in the organisation has a clear picture of what the cloud actually costs per application, per team, or per business unit.

The result is an invoice that runs higher than the data centre you left, with less flexibility than you expected. Not because cloud is too expensive. But because you never made the shift from ‘infrastructure in the cloud’ to ‘cloud-native architecture with cost accountability’.On-prem doesn’t fix that. It just moves the problem and adds five-to-seven-year hardware depreciation cycles to the books. The costs become slightly less visible, but you’ll feel them regardless. Especially as hardware prices keep climbing due to supply constraints.

KarmAI

For years, the architectural debt of lift & shift was a slow-burning problem. Systems ran, bills were higher than expected, but business carried on. AI has changed that. Not gradually, but abruptly.

AI workloads require infrastructure that is fundamentally different from what classic enterprise applications need. Data must be accessible easily and at scale, not locked in legacy storage structures that were once migrated one-to-one. Integration with managed services like Amazon Bedrockworks better when your environment is built on cloud-native standards. AI workloads also have a volatile cost profile: a training job or inference spike can consume more in hours than a classic application does in a month. Without FinOps mechanisms to monitor, cap, and allocate those spikes, you lose all control over your AI budget.

But that’s only the short-term pain. Because organisations that did lift & shift also migrated their data mess along with everything else. No proper classification, no clear ownership, no consistent structure. For classic applications, that was annoying but manageable. For AI, it’s fatal. You can’tfine-tune a language model on unstructured data scattered across poorly managed cloud systems. You can’t build a RAG architecture if you don’t know where your data lives or what its quality is.

Organisations that are serious about AI are being forced to revisit their cloud architecture anyway. What once looked like a technical problem has become a strategic delay… right at the moment AI is starting to make the competitive difference. And organisations that now decide to return to on-prem entirely will only widen that gap.

On-prem is capitulation

Organisations considering cloud repatriation today are asking themselves the wrong question. Because the discussion isn’t really about cloud versus on-prem. It’s about focus and resource allocation: what does your organisation actually need, and are you capable of managing it well?

Is it five to midnight for some organisations? Yes. But for those who commit to a cloud-native approach, it will become clear that cloud was the right choice all along. And that anyone thinking about repatriation today will find there’s still a great deal of cloud potential left untapped.

If the cloud hasn’t delivered on its original promises, it’s usually because the architecture never became truly cloud-native. Data that was never properly structured. FinOps discipline that was never built in: no centralised cost reporting, no shared accountability between IT and business, no culture where teams think about the cost of what they build and run. And a mindset that carried on-prem patterns into an environment where they don’t belong.

You don’t solve those problems by switching environments. You solve them by acknowledging them and addressing them. And that can absolutely be done in the cloud… if you’re willing to do it differently this time.

The post Back to on-prem? appeared first on Cloudar.

In the AWS ecosystem, MSPs take vastly different approaches when building customer-specific landing zones and cloud management platforms. Some approaches preserve your freedom and flexibility. Others quietly build the walls of a gilded cage.

The Two Paths: Open Standards vs. Proprietary Platforms

The Open Approach: AWS Landing Zone Accelerator (LZA)

AWS Landing Zone Accelerator represents the gold standard for customer independence. As an open-source solution built on AWS CDK and CloudFormation, we deploy LZA to provide several critical advantages:

• Complete transparency: All infrastructure is defined as code that you can read, understand, and modify
• No licensing fees: Open-source with no proprietary components
• AWS-maintained: Continuously updated by AWS to keep up to date with new services and features
• Industry standard configuration: With multiple documented sample configurations, you do not need to start from scratch.
• Full customer ownership: Deployed directly into your environment with complete access to the Infrastructure as Code
• Exit-ready from day one: If you ever want to manage it yourself or switch MSPs, you own your landing zone configuration

At Cloudar, we’ve built an entire landing zone practice around LZA precisely because we believe customers should never feel trapped. Your AWS foundation should be an asset you own, not a chain that binds you to any single provider.

The Proprietary Approach: Custom Orchestration Platforms

In contrast, many MSPs have developed proprietary cloud management platforms that create significant lock-in. These are often marketed as “revolutionary” or “next-generation” platforms that promise to make cloud management easier with “just a few simple clicks” or web-based portals that abstract away complexity.

The convenience is real. The long-term cost is hidden.

Here’s what proprietary platforms typically involve:

• Black box deployment: Resources are created through proprietary tooling that abstracts away the underlying infrastructure
• Dependency on custom APIs: Your operations become dependent on the MSP’s platform rather than native AWS tools
• Limited portability: Moving to another MSP or bringing management in-house requires re-platforming
• Knowledge gap: Your team never develops deep expertise in AWS native tools because they’re shielded by the abstraction layer
• Commercial leverage: The MSP knows that switching costs are high, affecting pricing negotiations and service quality over time

The Lock-in Mechanisms You Need to Watch For

1. Custom Landing Zones Without Source Code Access

Some MSPs deploy your resources using “their” landing zone—a pre-configured multi-account setup built with proprietary Infrastructure as Code that remains their intellectual property. When you want to leave, you inherit an AWS environment you don’t fully understand, configured by tools you can’t access.

The Cloudar difference: We can deploy LZA directly into your AWS accounts. Every CloudFormation stack, every configuration file, every security baseline—you have access to it all.

2. Web-Based Orchestrators That Become Operational Chokepoints

Fancy web portals that let you “deploy with one click” sound appealing. Until you realize that every operational change must flow through the MSP’s platform. Want to modify a VPC? You’re dependent on their UI. Need to adjust security groups? Better hope their platform supports your use case.

These orchestrators create operational lock-in: You can’t effectively operate your AWS environment without the MSP’s tooling. You’ve traded AWS complexity for MSP dependency.

3. “Simplified” Interfaces That Hide AWS Reality

Abstraction layers that promise to “make AWS easy” can create a dangerous gap between what you think you’re deploying and what’s actually running in your account. When problems arise—and they always do—you discover that your team doesn’t understand the actual AWS infrastructure because they’ve only interacted with it through the MSP’s simplified interface.

The Real-World Impact of Lock-in

Scenario 1: The Price Increase

Your MSP announces a 30% price increase. With an LZA you own and open standards, you have options: negotiate from a position of strength, bring management in-house, or transition to another MSP in months. With a proprietary platform, you’re looking at risky and arduous re-platforming work—and your MSP knows it.

Scenario 2: The Service Quality Decline

Your MSP gets acquired. The new parent company shifts focus, key engineers leave, and service quality drops. With an open approach, you can transition smoothly. With lock-in, you’re stuck enduring declining service while planning an expensive migration.

Scenario 3: The Strategic Pivot

Your company wants to build internal cloud expertise and eventually self-manage. With LZA, your team can learn standard AWS tools and practices from day one. When you’re ready to transition, you already have the skills and the code. With proprietary platforms, your team has learned the MSP’s tools, not AWS—setting your in-house capability building back by years.

Scenario 4: The Platform Limitation

Your business needs evolve, and you need to implement a complex AWS architecture that isn’t supported by your MSP’s platform. You’re now in the worst position: paying for a platform that constrains you, unable to use native AWS capabilities, and facing the choice between living with limitations or undertaking an expensive re-platforming project.

How to Evaluate Your Current or Prospective MSP

Ask these critical questions:

1. “What landing zone solution do you use?”
   • Red flag: “Our proprietary solution” or vague answers. Subscription based landing zones (yes they exist!).
   • Green flag: “your own AWS Landing Zone Accelerator” or “a per-customer AWS Control Tower with Customizations for Control Tower”
2. “What happens to our infrastructure if we terminate the contract?”
   • Red flag: Vague answers about “transition planning” or “it depends”
   • Green flag: “You keep everything—we’ll help with knowledge transfer, and you’ll have all the code and documentation”
3. “Will our team learn AWS-native tools, or primarily your platform?”
   • Red flag: “Our platform abstracts AWS complexity away”
   • Green flag: “We teach AWS best practices and native tools”

The Cloudar Philosophy: Your Cloud, Your Terms

Here’s what that means in practice:

Full LZA Implementation

Every customer gets its own AWS Landing Zone Accelerator, deployed directly into their accounts with complete source code access to the LZA configuration. Deployments happen in your account, giving you end-to-end visibility on your Landing Zone.

AWS-Native Tooling

We use CloudFormation, CDK, AWS Config, Systems Manager—tools that work with or without us. If you hire another AWS expert or build an in-house team, they’ll recognize everything immediately.

Comprehensive Documentation

You can read about every configuration option today, in the documentation published by AWS. So while we pride ourselves in sharing our knowledge, you are not dependant on us to explain what is going on

Additionally, we write customer-specific documentation  in our Confluence – from architecture decisions to operational procedures.. If you decide to leave, we can provide you with an export of that information.

Open Book Operations

You have full Read Only access to your AWS environment —we’re partners, not gatekeepers. Want to check our work? Go ahead.

Standard AWS Best Practices

We follow AWS Well-Architected Framework principles and industry-standard patterns. No “special sauce” that only we understand.

We succeed by giving you excellent service so you want to stay, not by making it painful to leave.

The Economics of Freedom

Some argue that proprietary platforms are necessary to provide better service or lower costs. We disagree.

Lower costs come from:

• Automation that scales across customers (which we use)
• Deep AWS expertise (which we have)
• Efficient processes (which we’ve refined over years)

Not from locking customers into proprietary platforms.

Better service comes from:

• Highly skilled engineers (which we continuously train)
• Customer focus (which our retention rate proves)

Not from proprietary abstraction layers.

We’ve proven that you can deliver excellent MSP services at competitive prices while keeping customers completely free. In fact, we believe customer freedom makes us better—we can’t coast on lock-in, so we must continuously earn our customers’ business.

Making the Right Choice

Before signing with any MSP, ask yourself:

• Do I understand what I’m getting into? Can you clearly explain how your infrastructure will be deployed and managed?
• What’s my exit strategy if things don’t work out? Is it measured in weeks, months, or years?
• Am I choosing this approach for the right reasons? Is convenience masking a lack of control?
• Do I retain full ownership? Of code, configurations, documentation, and knowledge?

Red Flags in MSP Sales Processes

Be wary if you encounter:

• Heavy emphasis on “simplicity” with little discussion of the underlying AWS architecture
• Vague answers about exit strategies and transition processes
• Marketing focused on proprietary platforms as the primary differentiator
• Contracts that grant the MSP exclusive rights to infrastructure code
• Lack of clarity about what you actually own vs. what you’re licensing

Conclusion: Freedom as a Feature

In the rush to cloud transformation, it’s easy to prioritize speed and convenience. And yes, a well-designed proprietary platform can deploy faster than custom LZA implementation—at least initially.

But cloud strategy isn’t measured in weeks. It’s measured in years and decades. The question isn’t “who can get me to cloud fastest?” It’s “who can help me build sustainable cloud capabilities that serve my business long-term?”

The MSP industry has a pattern: some providers build their business model around customer stickiness achieved through proprietary tooling. They create beautiful interfaces and slick demos that abstract away AWS complexity. Then, months or years later, customers realize they’ve traded AWS vendor lock-in for MSP vendor lock-in—often worse, because at least AWS is standardized.

At Cloudar, we reject this model fundamentally. We believe that customer freedom isn’t a bug to work around—it’s a feature to build for. We’re proud to be an AWS Premier MSP Partner that wins business through excellence, not lock-in.

Your cloud infrastructure is too important to be held hostage by convenient abstractions. You deserve an MSP that treats you as a partner who will grow and evolve, not a captive customer who might someday try to escape.

Choose partners who believe you should always have the keys to your own kingdom. Choose partners who succeed by being valuable, not by being necessary.

Choose freedom.

The post The Hidden Cost of Convenience appeared first on Cloudar.

Interestingly, Roles Anywhere does not completely solve the problem of not needing to create long-lived credentials, instead it moves (and reduces) the problem from credential management (rotation) to certificate management (distribution).

Setting up a Certificate Authority (CA) that can create certificates can be done with AWS Private Certificate Authority, and there are some options out there to manage distribution (usually as part of a bigger Device Management solution), but wouldn’t it be nice if the government of Belgium would do that for us?

The Belgian eID

In Belgium every citizens gets an identity card. The government makes sure that these get distributed and renewed, and since 2005 they contain a chip that holds (among other things) a certificate that can be used for digital authentication. If we can tie this to Roles Anywhere, we don’t have to worry about certificate management ourselves.

This eID is the size of a bankcard and the chip can be interacted with using off the shelf smart card readers. Most information can be read without any additional authentication – if you have the correct middleware, but to sign something you need to enter a PIN code.

The different certificates

Every eID holds an “authentication certificate” that is tied to the owner of the eID and has been signed by the “Citizens CA”.  The Citizens CA’s certificate is in turn signed by the Belgium Root CA . The Root certificate is self-signed. If we trust the Belgium Root CA, we will be able to validate any authentication certificate. More information about this setup can be found at https://repository.eidpki.belgium.be/#/home

The authentication certificate has two pieces of information that are going to be useful to us:

• The Issuer, showing us that the certificate is indeed issued by the Citizen CA.
• The Subject, containing the name and National Number (a number that uniquely points to one person) of the owner.

AWS IAM Roles Anywhere

Roles Anywhere allows you to use a certificate and its private key to get temporary credentials. You can think of it as an AssumeRole call, but instead of starting from an Access Key, we start from a certificate.

There are a few components we need:

• Configuration in our AWS Account, to indicate which CA we trust, and which roles can be assumed by different Issuer and/or Subjects. This also includes the creation of IAM Roles.
• Software on our computer to interact with the Card Reader so we can prove we have access to the eID and know its PIN.

For our use case we are going to trust the Belgium Root CA, and create a Role that can be assumed with any certificate issued by the Citizen CA. We will than limit the permissions of that IAM Role, so it can only write to a unique prefix in S3.

We can do that because every component of the Subject in our certificate gets mapped to a principal tag on the role session. E.g. if the subject is “C=BE, SN=Doe, GN=John, serialNumber=70010112345, CN=John Doe (Authentication)”, we have the following principal tags, and we can use principal tags in any policy.

• x509Subject/C: BE
• x509Subject/SN: Doe
• x509Subject/GN: John
• x509Subject/serialNumber: 70010112345
• x509Subject/CN: John Doe (Authentication)

Putting it all together

In our AWS Account

We will create the following resources in our AWS Account:

• An S3 Bucket
• A trust anchor, configured to trust the Belgium Root CA certificate that is publicly available via https://crt.eidpki.belgium.be/eid/brca6.crt.  This way we can use any certificate created by the Root CA of any of its subordinate CAs (like the Citizen CA)
• An IAM Role, where we configure the trust policy to only allow usage of the Role if:
  • The principal is the Roles Anywhere service (rolesanywhere.amazonaws.com)
  • The Issuer is Citizen CA (aws:PrincipalTag/x509Issuer/CN must equal “Citizen CA”)
  • The source is our trust anchor (aws:SourceArn equals the ARN of the Root CA trust anchor)
• An IAM Policy on this role that allows PutObject acces to all objects that start with the unique National Number (aws:PrincipalTag/x509Subject/serialNumber)
• a (Roles Anywhere) profile that ties the trust anchor to the Role. We could configure more settings here, like a mapping of fields from the certificate or a session policy, but don’t need to.

Of course we do not have to do this by hand, we create a CloudFormation template for this. It can be found in our sample repository: https://github.com/WeAreCloudar/cloudformation-samples/tree/main/templates/roles-anywhere-eid

On our machine

On our machine, we will need – besides the eID and its PIN (at least on macOS, where I tested this):

• a SmartCard Reader
• the Roles Anywhere Credential Helper (downloadable from the AWS Website)
• The AWS CLI (or any other tool we want to use with our AWS Credentials)

We also need a few pieces of information:

• The ARNs of the trust anchor, profile and role we created
• Our own National Number and the name of the bucket (to test our policy)

Ideally we’d be able to configure the credential helper to read any eID we insert, and this is possible if you install the Belgian eID Middleware using the following command.

aws_signing_helper credential-process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--certificate "pkcs11:model=Belgium%20eID;object=Authentication" \
--pkcs11-lib '/Library/Belgium Identity Card/Pkcs11/beid-pkcs11.bundle/Contents/MacOS/libbeidpkcs11.dylib

However because of an issue with the helper, this only works on macOS if you build the helper yourself from source. Since we do not want to do that, we can plug in our eID and find and identifier using “aws_signing_helper read-certificate-data”. You should get something like

Matching identities
1) bf013dd57aa34f8f9d4e22ab89dd8eb2 "SERIALNUMBER=70010112345,CN=John Doe (Authentication),C=BE,..."
2) 786f7572198a4a8a82ee7ef3f205d7be "SERIALNUMBER=70010112345,CN=John Doe (Signature),C=BE,..."


Copying everything between quotes, allows us to create a new file, selector.json that looks like this:


[
{
"Key": "x509Subject",
"Value": "SERIALNUMBER=70010112345,CN=John Doe (Authentication),C=BE,..."
}
]

and use that to select our specific certificate:

aws_signing_helper credential-process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--cert-selector file://selector.json

Our smart card reader will now prompt us for a PIN. If we enter the right one, we get credentials as output:

{"Version":1,"AccessKeyId":"ASIA...","SecretAccessKey":"COA...","SessionToken":"IQoJ...","Expiration":"2025-03-28T14:16:07Z"}


We can put this command in our ~/.aws/config file, so the AWS CLI will execute it for us, but for testing purposes it’s slightly easier to use the “serve” option.  The AWS Documentation has a longer explanation about how you can use the credential-process: https://docs.aws.amazon.com/rolesanywhere/latest/userguide/credential-helper.html#credential-helper-examples

Testing with s3

In one terminal we can run a credential / metadata server that can be used by the AWS CLI:

~$ aws_signing_helper serve -process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--cert-selector file://selector.json


This gives us
2025/03/28 14:28:12 Local server started on port: 9911
2025/03/28 14:28:12 Make it available to the sdk by running:
2025/03/28 14:28:12 export AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:9911/

We can than use a second terminal to run:

AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:9911/ aws s3 cp example_file s3://$bucket/serialNumber/70010112345/example_file

and get our file in s3:

upload: example_file to s3://.../serialNumber/70010112345/example_file

Conclusion

This experiment shows that being able to use an external device withIAM Roles, is a very powerful feature, because it can remove a lot of tasks related to certificate and credential management. Being able to do this with something I had in my backpocket made it extra cool.

However, the Belgian eID is probably not the best way to go about this in a real scenario:

• I just gave everyone living in Belgium access to my S3 Bucket. I could use a more scoped-down trust policy, but I would eventually run into limitations of trust-policy size.
• I also didn’t tackle Certificate Revocation Lists (CRL). Every time someone loses their wallet, I would need to invalidate certain certificates – this also runs into size limitations
• I already have a good way to give users access to AWS Accounts, using Identity Center or Cognito.
• The pkcs11 support on macOS seems limited, and I would have to install the middleware everywhere,

This does not mean that Roles Anywhere is not a good solution for this kind of cases:

• If you are already putting certificates on machines (e.g using an MDM solution), AWS IAM Roles Anywhere is a great fit
• Instead of relying on the Belgian Government you can buy an hardware key like a yubikey and use it as your certificate source. This is as secure as using the eID (only the machine with the key plugged in will be able to request credentials), and you would be limiting it to the exact serial number(s) in your trust policy anyway – making CRLs mostly moot.

The post Sign in with your eID: Using AWS IAM Roles Anywhere with a SmartCard Reader appeared first on Cloudar.

We were recently working on an AWS setup which involved a Network LoadBalancer (NLB) with a TCP listener and a requirement for sticky sessions. As we were seeing some strange behavior which we couldn’t immediately explain and which might be linked to the session stickiness we decided to make a small test setup.

The problem

Unlike an ALB where session stickiness is accomplished with cookies, the NLB uses a built-in 5-tuple hash table in order to maintain stickiness across backend servers. We access the NLB through its DNS name, which actually returns the IPs of the two NLB endpoints in a round-robin fashion with a TTL of 60 seconds.

We were looking for an answer on the following question: if our end-user would resolve the DNS and pick the IP of the first NLB endpoint to start the connection, the session will be routed towards one of the backend servers. But after 60 seconds the client could potentially re-issue the DNS query and start its connection with the other NLB endpoint. How will the stickiness and cross-zone loadbalancing behave? Will our end-user connection be routed to the initial server again, even if that means crossing AZ boundaries?

The situation

We started with a classical setup comprising of an NLB with an endpoint in each AZ and a Targetgroup having one instance as target in each AZ.

Scenario #1

• Cross-Zone loadbalancing: Disabled
• TargetGroup Stickiness: Disabled

How does it behave?

1. Client connects to the IP of the first NLB node: the connection is redirected to the server in AZ 1.
2. Client connects to the IP of the second NLB node: the connection is redirect to the server in AZ 2.

Since there is only one healthy target per AZ and cross-zone loadbalancing is not enabled, this situation results in ‘AZ-stickiness’: the traffic remains in the AZ in which it arrived. The setup relies on DNS to distribute client connections evenly across both NLB endpoints, but there is nothing to guarantee that a specific user connection is always directed to the same NLB endpoint, let alone to the same backend server.

Scenario #2

• Cross-Zone loadbalancing: Enabled
• TargetGroup Stickiness: Disabled

Allowing cross-zone loadbalancing and not requiring any stickness. This should give us complete randomness, shouldn’t it?

And so it does. We’re now completely randomized and hit every backend server, irrespective of our NLB endpoint ‘point of entry’. Works as expected.

Scenario #3

• Cross-Zone loadbalancing: Disabled
• TargetGroup Stickiness: Enabled

We’ve enabled stickiness on the targetgroup now, and disabled the cross-zone loadbalancing again. Let’s hope our client connection is now sticky to a specific backend server.

1. Client connects to the IP of the first NLB node: the connection is redirected to the server in AZ 1.
2. Client connects to the IP of the second NLB node: the connection is redirect to the server in AZ 2.

Ok wait, we’ve asked our TargetGroup to be sticky, but still our connection is balanced over both backend servers? What’s going on?

The fact that our NLB is not allowing cross-zone loadbalancing seems to prevent the connection from reaching the same backend every time. The connection enters via NLB endpoint 1 but stickiness has decided that the connection should go to server in AZ 2? Stickiness fails, the disabled cross-zone loadbalancing wins…

With only one healthy backend per AZ this behaves the same as not enabling stickiness at all. We’re pretty sure that with more than one backend per AZ the stickiness is maintained…within that AZ only. Interesting!

Scenario #4

• Cross-Zone loadbalancing: Enabled
• TargetGroup Stickiness: Enabled

Let’s solve this. We’ve enabled both cross-zone loadbalancing and targetgroup stickiness. We should hit the same backend server every time now.

And so it does. Only now we reach true stickiness and hit the same backend server every time, no matter how hard we try by entering via the loadbalancer node in the other AZ.

The conclusion

If you don’t allow cross-zone loadbalancing, then stickiness is only active within AZ boundaries. As DNS round-robin could direct a client to a different point of entry after the TTL has expired, strict stickiness is not guaranteed.

So if you really need stickiness to a specific backend target, you need to allow cross-zone loadbalancing (and live with the extra cost of inter-AZ traffic). Only now do the different loadbalancer nodes share the hash table of “client-to-target” stickiness.

Kinda logic, though…

PS: NLB idle timeout for TCP connections is 350 seconds. Once the timeout is reached or the session is terminated, the NLB will forget the stickiness and incoming packets will be considered as a new flow and could be loadbalanced to a new target.

The post Why AWS NLB stickiness is not always sticky appeared first on Cloudar.

The post 10 random things you probably didn’t know about Cloudar appeared first on Cloudar.

It’s a familiar scene: as you’re waking up or making your morning coffee, you hear the garbage truck roaring away outside when you suddenly realize your garbage bags are still rotting away in the garage. Do you run after the truck like a lunatic, even though it’s too late (of course I’ve never done this, but I’ve heard it’s a thing)? Or do you just let it go and let the garbage fester even longer? Either way, it’s a frustrating start to the day.

Luckily for me I live in the modern and very civilised province of Limburg of Belgium where the local government has made a website available with a downloadable calendar showing all dates the garbage truck comes along: so handy! They even provide the calendar in a number of digital formats, including iCal. I could just import this into my phone and set an alert on it, but where’s the fun in that? After some fiddling around or “reverse engineering” as they call it, I found out that the data for the calendar files was provided by an undocumented but publicly available API.

Being the sucker for coding up and fooling around with APIs that I am (and given my state of utter boredom while under full lockdown in Belgium) I decided to code up a project based on this API that would send me an SMS notification the evening before garbage collection. Because that’s my idea of fun. And that way I would be reminded to take out the trash on time and keep my garage clean.

Want to know this neat trick to save you from rubbish resentment? Read on!

So for this project I partly used the AWS SAM framework and set up a CodePipeline to deploy my code changes instantly on AWS (stay tuned for a future blogpost on this).

First let’s code up the AWS SAM template file containing the necessary resources for this project.

The resources I used are:

• Type: AWS::Events::Rule
  A Cloudwatch event rule to invoke a lambda each evening around 8.
• Type: AWS::Lambda::Permission
  To give the Cloudwatch event rule permissions to invoke the lambda.
• Type: AWS::Logs::LogGroup
  A log group for the lambda logs.
• Type: AWS::Serverless::Function
  The lambda that will run the show.
• Type: AWS::IAM::Role
  An IAM role for the lambda to have the necessary rights like accessing SNS for the SMS notification.

Next we code up the lambda that will call the API and send us an SMS.

The API provides event data for each month by just requesting the year and month in the url path like this ‘/2020-03’. So we can easily get the data for the current month and loop through its events, checking whether any are occurring the next morning. If so we receive an SMS notifying us (note that you can provide the number of your girlfriend here, but it’s not recommended as I have found out it isn’t that effective).

And that’s all folks!

I’m aware that this trick is no rocket science, and that this isn’t even a full guide to get to the same result (leave a comment if you want some more details), but I thought it would be fun to share how I resolved one of life’s little problems using AWS.

As I’m moving toward two years of professional experience using the platform, I’m still amazed at how easily you can code some services together and create a working, scalable and highly available project in a matter of minutes or hours.

Hope you liked my trashy tale – keep your garages clean and your heads in the cloud!

The post Let’s trash talk! appeared first on Cloudar.

We will use a custom resource written in Python that will be able to create ACM certificates with DNS validation. The custom resource will also automatically validate this certificate if the validation domain is managed by a Route53 hosted zone. We will also be able to specify an AWS region to create the certificate in, this region is independent of the Cloudformation stack region which for example makes it possible to deploy a certificate in region us-east-1 (to use with cloudfront) while deploying the stack in region eu-west-1. The resource will also provide the certificate arn as an output parameter so it can be used by other resources in the stack. Lastly when you delete the custom resource it will cleanup all validation records and the certificate itself.

Requirements:

• Python3
• Pip
• Bash
• Zip
• An S3 bucket to deploy the custom resource package on
• A hosted zone for the validation record

Implementation:

Let’s get started by downloading all the required code from our GitHub repository.

Step1: Uploading the custom resource package

In this step we are going to prepare the custom resource package and upload it to an S3 bucket.

First we go into the custom resource directory.
cd cloudar-acm-plus-custom-resource

Next we execute a script to install all required dependencies.
sh install_dependencies

Now we are ready to create the package.
sh pack_custom_resource

You will now find the zipfile ‘cloudar-acm-plus-custom-resource.zip’ in ‘cloudar-acm-plus-custom-resource/packed’, upload this zip file to your S3 bucket.

Step2: Creating a Cloudformation template

Now we can create a Cloudformation template in which we use this custom resource to create an ACM certificate.
You can use the template ‘cfn.yaml’ as an example.

First create a Lambda resource as following

Use the name of your bucket for the property ‘S3Bucket’ .

Next we create the custom resource.

We can set the following properties here:

• DomainName: (REQUIRED type:String) The domain name for the acm certificate.
• AdditionalDomains: (OPTIONAL type:List) Additional domains for the acm certificate
• ValidationDomain: (REQUIRED type:string) The domain name for the validation domain of the acm certificate
• HostedZoneId: (REQUIRED type:string) The hosted zone id for the validation domain of the acm certificate
• CertificateRegion: (REQUIRED type:string) The region to deploy the acm certificate in
• IdempotencyToken: (REQUIRED type:string pattern: \w+) The idempotency token for the create call of the acm certificate doc: https://docs.aws.amazon.com/acm/latest/APIReference/API_RequestCertificate.html#ACM-RequestCertificate-request-IdempotencyToken
• CertificateTags: (OPTIONAL type:list) The tags for the acm certificate

In order for the DNS record cleanup and delete certificate functionality to work when you delete the Cloudformation stack it is important to set the following output.

As you can see we can access the arn of the certifcate created by the custom resource with the GetAtt function on the resource.
!GetAtt CreateCertificateCustomResource.certificate_arn

Step3: Deploy the cloudformation

Finally the only thing left to do is deploy the Cloudformation template.
Once the deploy is started Cloudformation will create the Lambda containing the code from step1 and start a custom resource which will create the certificate and validation records. Once the status of the certificate becomes ‘ISSUED’ the custom resource will finish successfully and report the arn of the certificate back to Cloudformation. We can now further use this arn in other resources in the Cloudformation template.
When you delete the Cloudformation stack, the custom resource will cleanup the validation records in the hosted zone and delete the certificate.

CREATE_COMPLETE

The post Validate ACM certificates in Cloudformation appeared first on Cloudar.

Video isn’t the best reference material if you want to quickly find one of those tips, so I will add a quick summary of them in this blogpost.

• By specifying source_profile in your .aws/config file, you can tell the AWS CLI to use credentials from one profile to assume the role_arn you configure in another profile. This also works in a chain of profiles and can include asking for an MFA token.
• You don’t have to get credentials from the .aws/credentials file, you can tell the CLI to get them from environment variables (this also works if you do not specify anything), instance metadata, or the ECS container role.
• There is a --debug flag, which gives you a lot of output to see everything that’s happening behind the scenes.
• In most cases you do not want that full debug output. Enabling history and using aws history show will give you the most relevant parts in an easy to read format.
• The CLI has the annoying habit of resolving all urls you give it. You can disable this with aws configure set cli_follow_urlparam false.
• You can make S3 uploads and downloads go a lot faster by tweaking the max_concurrent_requests. Or make it not exceed a certain speed with max_bandwith. See here for all options.
• The cli supports defining your own aliases. Look in this awslabs repository for examples.
• Instead of relying on an external tool (that needs to be installed), you can get specific fields from the response by using --query. Combine this with --output text to strip json-specific syntax.
• You can use completion in different shells.
• There are waiters you can use directly from the command line, so you do not have to write your own check-sleep loops in bash.
• Other useful cli tools include cfn-lint, aws-encryption-sdk, aws-shell and aws-vault
• There is a version 2 coming.

The post $ aws help # things you might not know about the AWS CLI appeared first on Cloudar.

In this blog we will use AWS Rekognition to search for ourselves in a collection of pictures.

First we need to acquire this collection, I simply used a Chrome extension to download all pictures of a specific Facebook album.

Next, we will create an S3 bucket to store the pictures, because this project reminds me a bit of where’s waldo I will name the bucket ‘awesome-waldo-bucket’.

Now we will upload all the pictures to this bucket using the command.

aws s3 sync . s3://awesome-waldo-bucket

Enjoy the terminal doing its job, casual snap some screenshots to non-IT friends. It scores well. #hackerMan

While the pictures are uploading we will create the AWS Rekognition collection with the following command.

aws rekognition create-collection –collection-id awesome-waldo-collection

After we create our collection and the pictures are uploaded we will use a Python script to add faces to the collection. The script will loop all the pictures in the S3 bucket and send them to the Rekognition collection with the ‘index faces’ operation, when a picture is added to the collection AWS will automatically process it and detect the faces, these faces will be stored in a map which we will be able to query later.

AWS documentation on this operation.

I added a hard-coded counter in this script just to follow progress.

import boto3
from boto.s3.connection import S3Connection

conn = S3Connection()
bucket = conn.get_bucket('awesome-waldo-bucket')
count = 1

for key in bucket.list():
    print("Added " + str(count) + " pictures out of 1697...")
    count = count + 1

    bucket='awesome-waldo-bucket'
    collectionId='awesome-waldo-collection'
    photo=key.name.encode('utf-8')
    client=boto3.client('rekognition')
    response=client.index_faces
    response=client.index_faces(CollectionId=collectionId,
                                Image={'S3Object':{'Bucket':bucket,'Name':photo}},
                                ExternalImageId=photo,
                                QualityFilter="AUTO")

Next execute the Python script.

You will find that it takes a very long time to process this number of pictures. For the sake of keeping it simple we did not do any batch processing in this blog but it is possible.

AWS documentation for batch processing.

After the images are done processing find a clear and recent picture of yourself, name it ‘me.png’ and upload it to the S3 bucket.

Be sure that it is a clear picture of your face and that you are the only one in it.
For example.

Now that we have all elements in place we can use the following command to retrieve faces matching our sample.

aws rekognition search-faces-by-image –image ‘{“S3Object”:{“Bucket”:”awesome-waldo-bucket”,”Name”:”me.png”}}’ –face-match-threshold 70 –collection-id “awesome-waldo-collection”

You can play with the ‘—face-match-threshold’ to get more or less correct results. The command will return json containing data on the matched faces.

You can repeat this with different sample pictures of your face to get more results.

And now the moment of truth, as it turns out I was not that popular with the photographers on the festival this year as I’m only in two pictures. Better luck next year!

As you can see, AWS Rekognition is a great service to find yourself in a picture even in a big crowd!

This picture surprised me the most, even when your face is very blury AWS Rekognition can still find you! What a gift of technology indeed!

I hope this blog has given you some insight in the possibilities with AWS Rekognition and wish you the best of luck in finding yourself!

The post Discover yourself appeared first on Cloudar.