When companies make an IT decision, they look at licensing costs, implementation costs, sometimes training costs. What they rarely calculate is what we’ve jokingly come to call the sigh surcharge. It’s the sum of human friction that arises when systems aren’t set up optimally. It’s the second of hesitation before a too-slow action. It’s the workaround everyone in a company knows about but nobody has actually fixed. It’s the copy-paste from one system to another because the integration never happened. It’s the mental weight of working against your tools instead of with them.

A single sigh seems trivial. Collectively, it’s the unspoken productivity leak that drains away quarter after quarter. Fortunately for most companies, it’s the one surcharge that never shows up on an invoice. But imagine if it did? That, just like VAT, a sigh surcharge got added to every software decision. A percentage on top of the implementation cost reflecting the expected human friction. Poorly designed interface? +15% sigh surcharge. No integration with existing tools? +20%. Slow load times at critical moments? +10%. Illogical workflows that clash with how people actually work? +25%. Suddenly the cheapest option looks a lot more expensive. And at that point, the solution that was actually implemented properly – with thoughtful workflows, meaningful automation, and a user experience that doesn’t get in people’s way – starts to look a lot more appealing.

The sigh is a signal

Every sigh is really a data point. It’s feedback nobody asked for but that’s given constantly. It says: this isn’t right, this isn’t working. This costs me more energy than it should. In organizations where the technology works well, you barely hear that sigh. Not because people stop sighing about other things, but because the tools simply do what they’re supposed to do. To some, that might sound like a luxury. To us, though, it’s the baseline that people deserve from the systems they work with every day.

And fortunately, you don’t need to be a data scientist to interpret these data points. The first step is simple: head over to the coffee machine. Stand there for a few hours and find out what people complain about most. They’ll point you straight to the biggest source of frustration. A second step is to look within the organization for the biggest workaround-builders — the people who, for every decision, already have a plan B or C in mind. In IT terms: look at all those Excel sheets patching a “gap” somewhere, because no automated process exists. Or all the emails that could really just have been a notification. They’re disguised examples of a sigh. And once you add up all these moments and multiply them by the number of people frustrated by the same bottleneck every day, the sigh surcharge starts to turn into a number you can actually put on a slide.

At Cloudar, we believe technology works best when it’s invisible. When people can simply do what they’re good at, without having to think about the tools that are supposed to help them. That doesn’t always mean choosing the most expensive option. It does mean making the right choice and executing it well. Because a cloud solution that’s only half implemented, or whose configuration doesn’t match how people actually work, doesn’t deliver value. It delivers sigh surcharges. The real return on investment of a good implementation isn’t just speed or cost savings. It’s, above all, a sound you no longer hear on the work floor.

The post The sigh surcharge appeared first on Cloudar.

AWS: the choice that determined everything 

In the cloud industry, “focus” is a dangerous word. It sounds great in a pitch deck, but in practice it means leaving opportunities on the table. Customers running on Azure, projects you have to refer elsewhere, RFPs where you know from the start you won’t win. We accepted that over the years. Our drive was (and still is) simple: by concentrating everything on one platform, we can become the best possible partner on it. Not broad, but deep. That translates concretely into certifications, into architectural knowledge, into the relationships you build with the AWS team… and ultimately into the trust that customers place in you for their most critical workloads. Becoming an AWS Premier Partner was a milestone in that journey. But it was never the endpoint, let alone the ultimate goal. 

With the rise of generative AI, the role of cloud architecture has fundamentally changed. The question is no longer “how do we migrate our workloads to the cloud?” but “how do we design infrastructure that can run AI models at scale, securely, cost-efficiently and with the right governance?” That is an architectural challenge of an entirely new order. Inference costs, latency optimisation, data residency, IAM configurations compatible with foundation models, VPC setups that withstand enterprise compliance: these are the problems our customers face. And these are the problems for which you need to know AWS to the bone. 

AWS Bedrock has become the platform in that landscape where enterprise AI adoption takes place in practice. Not because it is the only option, but because it is the option that takes security, compliance and integration with existing AWS workloads seriously. For organisations that have been running on AWS for years, Bedrock is the most logical bridge to production-ready AI. 

What Anthropic Authorized Reseller status concretely means 

I’ll be honest: it took quite some effort. The Anthropic Authorized Reseller status is not a badge you apply for and receive in the post a week later. It requires knowledge, skill, a demonstrable pipeline and navigating an approval process in which Anthropic itself decides whether you clear the bar. A handful of AWS partners worldwide manage that. Most do not. And we are now the first local Benelux player to join that group. 

What this concretely means for the Benelux market is perhaps less well known than it should be. Since October 2025, AWS requires that a customer’s managing partner be officially recognised by Anthropic, if that customer wants access to the latest Claude models via Bedrock. This applies to Claude 4, Claude Code, and all future frontier models. Without a recognised partner: no access. 

In practice, this meant that large organisations in the Benelux were faced with a closed door for months. They wanted to experiment with the most capable models on the market, had the budget, had the use cases, but had no AWS partner with the right status. For CTOs, CIOs or enterprise architects who take AI seriously, there are a few practical implications worth mentioning. 

• First: access to frontier models has become a partnership question, not just a technical or commercial one. The architectural choices you make today – which AWS partner manages your account, how your Bedrock setup is configured – partly determine which AI capabilities you can deploy tomorrow. 

• Second: enterprise seats for Anthropic are not the same as API access. It is about onboarding, governance, user management and monitoring data flows within your own AWS environment. That requires someone who knows both the Anthropic side and the AWS infrastructure. 

• Third: Claude Code is a relevant example of where this is heading. AI-driven code assistance running directly on your own AWS infrastructure, without your code leaving the perimeter of your own environment. That is a different proposition from a SaaS tool with a subscription. 

In 2014 we chose AWS because we believed that focus leads to expertise, and expertise leads to trust. In 2025 and 2026, AI has not changed our conviction. The complexity of production-ready AI at enterprise scale requires precisely the kind of deep platform knowledge we have invested in over the past twelve years. 

The Anthropic Authorized Reseller status is, in that light, not just a commercial milestone. It is the validation that our approach works. And for our customers, it is the guarantee that they can get started with the most advanced AI models on the market, guided by a local team that knows the full stack: from infrastructure to model deployment to governance. But as has been said before, for us this is far from the final destination. 

The post Blood, sweat and tears appeared first on Cloudar.

It is the most primitive cost optimization imaginable, and the fact that people reach for it tells you exactly where AI operations are heading. Developers are willing to make their assistant grunt like a Neanderthal to trim a few thousand tokens off a session. Something changed to make that feel worth doing.

For the past two years, most of us have been running AI workloads on someone else’s dime. Flat-rate subscription plans made generative AI feel free at the point of use. Fire off as many prompts as you want, let your coding agent churn through refactors overnight, and the bill stays the same.

That era is ending. LLM providers are increasingly pushing heavy workloads toward usage-based pricing, credits, stricter rate limits, or token-metered APIs, and the reason is simple: the plans were too cheap. The compute behind a heavy agentic workload costs real money, and providers can no longer subsidize power users with the margins of light users. Rate limits are tightening, “unlimited” tiers are disappearing, and token-based billing is becoming the norm.

If you have ever watched an AWS bill balloon because nobody rightsized their EC2 fleet, you know exactly what happens next. The token is the new gigabyte, and we are about to relive the FinOps learning curve, this time for AI.

That is why we need FinAIOps: FinOps for AI systems, where LLMOps meets cost discipline in production. It is the practice of running AI workloads with the same operational and cost rigor we apply to everything else in the cloud.

Lessons from building agents in production

At Cloudar we have been building and running agentic AI workloads in production. Doing that teaches you very quickly where tokens go to die. These are the practices that made the biggest difference.

1. Use smaller models for simpler tasks

Not every task needs your most capable model. Agent routing, classification, and simple extraction run perfectly well on smaller, cheaper models. Reserve the frontier model for the reasoning-heavy steps. The per-token price difference between a frontier model and a small model on Bedrock is easily an order of magnitude, so a lightweight router that dispatches work to the right specialist pays for itself immediately.

This is where Amazon Bedrock shines: switching between model families and sizes is a configuration change, not a re-architecture. You can experiment with the cost and quality trade-off per task and measure the result.

2. Gatekeep the agent: deterministic first

The cheapest LLM call is the one you never make. If a task can be solved deterministically, solve it deterministically. Fetching a metric, checking a threshold, parsing a known log format: none of that needs an agent. Put a gate in front of your agent that handles the predictable cases with plain code and only escalates genuine ambiguity to the model.

3. Keep prompts compact

Prompts are tokens, and tokens are money. Every instruction, every example, every “please be helpful and thorough” costs you on every single invocation. Ruthlessly edit your system prompts. Say what you need, cut what you do not, and test whether shorter prompts degrade quality. Usually they do not.

4. Limit the tools your MCP servers expose

This one is underestimated. Every tool definition you expose to an agent is injected into the context on every call. It might feel convenient to give your agent the full toolbox, but a wall of tool schemas eats tokens before the agent has done any actual work.

Only expose the tools the agent genuinely needs. And when you do not know upfront which tools an agent will use, for example with open-ended investigative workloads, take an iterative approach: expose everything, have the agent write a short analysis after each run, and after a set period feed those results to an advanced model to analyze which tools earn their place and which can be dropped. Treat your tool catalog like an IAM policy: least privilege, reviewed regularly.

5. Cap the loop

Agentic workloads can run away. An agent stuck in a retry loop is the token equivalent of a Lambda retry storm. Set a maximum number of steps and tool calls per run, and a hard token budget per invocation. Fail loud, not expensive.

6. Mind your outputs and your history

For many frontier models, output tokens are priced significantly higher than input tokens. Ask for structured output instead of prose, set max_tokens deliberately, and instruct the model to be terse. And do not drag the full conversation history through every turn: summarize or window older context, because in long-running agents the context snowballs and you pay for all of it on every call.

7. Use prompt caching

On Bedrock, prompt caching lets you cache the static parts of your context, such as system prompts and tool definitions, and pay a fraction of the input price on subsequent calls. If you have a large tool catalog you cannot trim further, caching softens the blow considerably. Combined with point 4, this is one of the highest-impact optimizations available today.

8. Batch what is not urgent

Amazon Bedrock batch inference runs asynchronous workloads at a 50 percent discount compared to on-demand pricing. You submit a JSONL file with your prompts, Bedrock processes them asynchronously, and the results land in S3, typically within 24 hours. Periodic analysis jobs are a textbook example: nobody is waiting on the result, so there is no reason to pay real-time prices for it.

Bedrock also offers a Flex service tier for supported models, trading latency for lower cost. Unlike batch, Flex uses the regular invocation API: you add a service tier parameter to your call and accept increased latency in exchange for the lower rate. Availability and discount levels depend on the model and tier, so check the current pricing page before assuming the same economics as batch. It is a good fit for background agent runs that are real-time in shape but not in urgency.

The practices above, at a glance:

Measure it or it did not happen

Optimization without measurement is guesswork. Two AWS capabilities matter here.

Amazon CloudWatch gives you deep visibility into every dimension of your Bedrock usage. Bedrock publishes runtime metrics to the AWS/Bedrock namespace, with ModelId among the available dimensions. Which dimensions apply, and whether ModelId alone is enough, depends on the model, Region, service tier, and whether you invoke through an inference profile, so treat any dashboard as a starting point rather than a template that works unchanged everywhere. The ones to watch:

• Invocations: how often each model is called
• InputTokenCount and OutputTokenCount: where your money actually goes
• InvocationLatency: the quality side of the trade-off
• InvocationThrottles and InvocationClientErrors: the signals that an agent is misbehaving

Build a dashboard on these, and set an alarm on token consumption. For example, a CloudWatch alarm on the hourly Sum of OutputTokenCount per model catches a runaway agent loop within the hour instead of on next month’s invoice:

aws cloudwatch put-metric-alarm \
  --alarm-name bedrock-output-token-spike \
  --namespace AWS/Bedrock \
  --metric-name OutputTokenCount \
  --dimensions Name=ModelId,Value=<your-bedrock-model-id-or-inference-profile> \
  --statistic Sum --period 3600 \
  --threshold 5000000 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:eu-west-1:123456789012:ops-alerts

Replace the model ID with the exact model or inference profile dimension used in your account, since Bedrock model IDs are versioned and vary by Region and provider. Enable model invocation logging as well, so you can see the actual prompts behind an anomaly. Treat anomalous spend like any other operational incident.

Bedrock application inference profiles let you tag usage per agent, per workflow, or per customer. You create a profile that wraps a foundation model, attach cost allocation tags, and invoke through the profile ARN instead of the model ID:

aws bedrock create-inference-profile \
  --inference-profile-name customer-a-investigator \
  --model-source copyFrom=arn:aws:bedrock:eu-west-1::foundation-model/anthropic.claude-sonnet-4-5 \
  --tags key=customer,value=customer-a key=agent,value=investigator

Those tags flow through to Cost Explorer and CloudWatch, turning “the AI bill is high” into “agent X on task Y for customer Z is driving the cost.” For a managed service provider, this is essential: it enables proper chargeback and shows customers exactly what their AI workloads cost.

Finally, track the metric that actually matters: cost per outcome. Tokens per completed task beats tokens per month. An agent that uses three times the tokens but delivers twice the results is the cheaper agent. That is the KPI that makes FinAIOps a business discipline instead of a savings exercise.

The bill is coming. Be ready.

The shift from plans to tokens is not a pricing tweak, it is a forcing function. Teams that treat AI as free will get the same surprise bills we saw in the early cloud days. Teams that apply FinAIOps discipline, right-sized models, gated invocations, lean prompts, curated tool catalogs, caching, batching, and real cost attribution, will run AI workloads that are both powerful and predictable.

We learned the hard way with EC2 that capacity discipline pays off. Let us not learn it the hard way again with tokens.

The post FinAIOps: Why Token Economics Will Define Your AI Operations appeared first on Cloudar.

When Broadcom closed its acquisition of VMware, everyone in the industry knew something was going to change. Have we already forgotten what happened to Symantec, among others? Broadcom bought Symantec’s enterprise unit in 2019, narrowed it to around 2,000 of the largest accounts (the Global 2000), and raised prices on most of the rest. VMware is the same script on a bigger stage. The only surprise was how fast it came. In short order, perpetual licenses were scrapped, thousands of cloud partners (Cloudar among them) received a termination letter, and a catalogue of over 8,000 SKUs was reduced to two bundles (VMware Cloud Foundation and VMware vSphere Foundation). The shift from per-CPU-socket licensing to per-physical-core licensing also meant that organizations expecting to simply carry on with an existing installation were suddenly facing invoices two to five times higher.

But those cost increases are relative. The real impact only lands on 11 October 2027, when general support for VCF 8 ends. By that date, every VMware customer has to either migrate to VCF 9 at the new (read: higher) prices and under the new model, or be shown the back door and leave the platform behind.

Not deciding is also a decision

Big decisions like this get postponed fast. Or, in the alternative: you do not decide, and you take the hit. I understand the reflex. Migrations are complex. Infrastructure is critical. You do not want to take risks with production environments. And maybe you are thinking Broadcom will course-correct once the pressure builds. I do not want to give you false hope. By now there is plenty of evidence that Broadcom has no intention of changing course. The messaging is consistent, prices have not come down, and contract negotiations follow a tight, inflexible script.

More importantly, the migration window shrinks every month. A phased, carefully prepared migration of enterprise workloads takes two to three years in practice. If you only start evaluating in 2026, you have almost no room for a calm, controlled execution. The result of waiting too long is not that you end up with more options. It is that you are forced to choose between an expensive rushed migration and an extension with Broadcom at prices you do not actually want to pay. Everyone wins, except the end customer.

And if you are going to migrate anyway, why not move to another on-premises hypervisor? It might look like the most logical step. Nutanix, Proxmox, Microsoft Hyper-V: there are alternatives that keep the virtualization layer intact.

My answer is a nuanced one. For certain workloads with strict data-residency requirements or specific latency constraints, on-premises can remain relevant. But for most companies, switching to another hypervisor is only a stopgap. You solve the price increase, but you keep working with the same operational model. You still manage hardware, license renewals, capacity planning, and the staffing overhead that comes with all of it. On top of that, you trade one vendor dependency for another.

Staying on-premises carries a second problem beyond the licensing model. Hardware is becoming a scarce commodity, and so is datacenter floor space. Prices are going through the roof, and supply is no longer guaranteed.

AWS offers something structurally different: the option to move away from the virtualization layer over time. And getting there is no longer a leap of faith. If you want the smallest possible change on day one, Amazon Elastic VMware Service (Amazon EVS) runs VMware Cloud Foundation directly inside an Amazon VPC in your own AWS account, so the same vSphere, vSAN and NSX stack your team already operates lands on AWS without a re-architecture. If you would rather leave the hypervisor behind right away, AWS Transform MGN (formerly AWS Application Migration Service) replicates your virtual machines block by block and cuts them over to Amazon EC2 with minimal downtime, while AWS Transform for VMware automates the unglamorous parts: discovery, dependency mapping, network conversion, and right-sizing the target instances. The two paths trade off differently. Amazon EVS keeps your VMware stack intact, and with it the VMware licensing, but moves it off your own hardware and onto AWS as the fastest possible landing. The MGN route leaves the hypervisor, and the VMware license, behind outright. Either way your teams keep their existing way of working and do not have to relearn everything on day one.

That is the floor, not the ceiling. Once a workload runs on AWS you can modernize it on its own timeline, and only where the business case justifies it: move a self-managed database onto Amazon RDS or Amazon Aurora, lift a tier of servers into containers on Amazon ECS or Amazon EKS, replace a scheduled job with an AWS Lambda function, and reach for managed AI through Amazon Bedrock. None of that is realistic to run well on-premises, or it is simply too expensive to operate yourself. A like-for-like hypervisor swap does not get you here: it can lower the licensing bill, but the operational model and the eventual modernization are left untouched. Moving to AWS is the option where the immediate fix and the longer-term path share one destination.

We have been doing this at Cloudar for 12 years, and a lot of those conversations reveal the same pattern. At some point an organization made a sound technology choice. And today they find that the choice is holding them in a situation they can no longer defend, or no longer want to. VMware was an excellent choice for years. It was stable, well documented, broadly supported. But the lesson of the Broadcom acquisition is that vendor lock-in is not an abstract technical concern. It is a concrete business risk that, sometimes only after ten or fifteen years, shows up as something you simply have no alternative to.

Moving to AWS does not make that problem disappear entirely, of course. Every major cloud provider has its own ecosystem and its own logic. But the nature of the dependency changes fundamentally. In the cloud you pay for what you use, and you can decide workload by workload how and where you run it. The technical architecture of cloud-native services (open standards, open containers, open APIs) lets you switch when you want. That is structurally different from proprietary hypervisor software with a binary licensing logic. It lets you design with a solid exit scenario built in from the start.

Start with insight, then migrate in phases

Migrations rarely fail on the technology. They fail through a lack of preparation, unclear priorities, and too little attention to the human side of the process. The first step every organization should take is insight: which workloads are you running, what are the dependencies, and what is the real cost of your current VMware installation versus the alternatives? Only once that is clear can you make a rational decision. The tooling to get there has matured: AWS Transform for VMware can inventory a vSphere estate and map its dependencies automatically, and AWS Migration Hub tracks the moves once they start. A Migration Readiness Assessment that used to take months is now achievable in a few weeks.

This is where AWS has invested most visibly. AWS Transform, its agentic migration service, puts AI agents on the parts of a migration that used to eat months of consultant time: discovery, dependency mapping, wave planning, and network conversion, with the actual rehost driven through AWS Transform MGN. It is not VMware-only either; the same agents cover Windows and .NET modernization and mainframe workloads. AWS reports discovery that once took a quarter compressing to about a week, and landing-zone networking provisioned up to 70 percent faster. None of that removes the need to start early. It does mean the preparation that scared you off a year or two ago is no longer the bottleneck it was.

AWS can help fund a lot of this work through the Migration Acceleration Program (MAP). The assessment phase produces a workload inventory, a dependency map, and a costed business case, all of which are yours to keep, whichever direction you choose afterwards. For projects with a strong enough case, MAP can also offset a meaningful share of the migration cost itself. A migration partner runs the assessment and unlocks that funding on your behalf, which is the role we play at Cloudar.

The approach we recommend is phased. Start with the most portable workloads on Amazon EC2, which immediately eliminates the VMware license cost, then replatform workload by workload to AWS-native services as the business case justifies it. And in the meantime, put all new VMware deployments on ice. Every new workload you still put on VMware today is a workload you will have to migrate again later.

What I want to leave you with is this. The best migrations are the ones you carry out when you have the time and the space to do them well. You still have that time and space now, but the clock is ticking. As I write this, there are 467 days left until 11 October 2027. That may sound like a lot…

The post Leaving VMware is a question of timing appeared first on Cloudar.

That is the uncomfortable part of putting an LLM-powered agent into production: you inherit a dependency that never shows up on the architecture diagram. Someone else’s model has to stay reachable for your system to do anything at all. Usually the ways it fails are mundane. The model gets throttled the week you need it most, or deprecated on the vendor’s timeline rather than yours, or it simply has a bad afternoon while your incident volume spikes. June was the reminder that it can also be a regulator drawing a line. Either way, the day it happens, your “autonomous” system is exactly as autonomous as a 500 error.

We had been building an operations agent that investigates tickets and reasons over live infrastructure, and we had already decided not to hard-wire it to a single model. The June suspension is what turned that from a prudent hedge into an obvious one, and it is why we are writing this up now. The agent is genuinely useful, which is the problem: the more people rely on it, the less acceptable “the model is unavailable right now” becomes as an answer. This post is how we think about that availability problem, why we landed on Amazon Bedrock as the foundation, and (more importantly) the things we got wrong on the way, because the interesting lessons are in the pitfalls, not the happy path.

Two ways to stop betting the business on one model

There are really only two structural answers to “what happens when my model is gone”:

• Configure fallbacks. Have more than one model the agent can run on, and fail over when the primary is unavailable.
• Run your own model. Remove the third-party API from the critical path entirely, so availability is something you control rather than something you subscribe to.

Both are sound. Both are also more subtle than they sound, and the subtlety is the whole point of this article. Neither is “set a second model ID and you’re done.”

The reason we built on Amazon Bedrock is that it makes both strategies reachable from one place. Through a single AWS IAM surface, one set of credentials, one regional endpoint, and one billing path, Bedrock gives you access to a large catalogue of foundation models from multiple providers. Via Amazon Bedrock Custom Model Import you can bring your own open-weight models, and Amazon Bedrock Marketplace adds a catalogue of others to deploy. You can layer Amazon Bedrock Guardrails across all of them as a provider-independent safety control, and you keep your data inside your chosen AWS Region. For a European services partner like us, whose customers are specific about where their data is processed, Region control often settles the whole approach before model quality even enters the conversation.

So far, so much like a Bedrock landing page. Here is where it got real for us.

Pitfall 1: “swap the model ID” is an access abstraction, not an application one

The single most useful mental model we developed is this distinction. Bedrock abstracts access and transport: one credential, one API, one bill, to reach many models. It does not abstract the request and response shape your application actually depends on.

When you start, you almost certainly reach for a provider’s own SDK or the request format of whichever model you adopted first. That code grows roots. Your agent loop parses the specific content-block shape that model returns: the way tool calls are represented, the field names, the IDs. Your retry logic, your streaming handling, your token accounting all quietly assume one family’s conventions. The moment you point that code at a different model family, the transport works perfectly and the parsing falls apart, because you didn’t insert a model-agnostic layer. You inserted one-vendor-on-Bedrock, which borrows Bedrock’s plumbing but speaks only one dialect.

Amazon Bedrock does offer a genuinely cross-model interface, the Converse API, which normalises messages and tool configuration into a common shape across many models. Most current foundation models on Bedrock are reachable through it, so adopting Converse from day one avoids a lot of this pain. The catch is at the edges: you trade away some of the richer, provider-specific surface, and a few models still expose their newest capabilities only through their own API shapes. A fully general agent can therefore end up maintaining more than one transport path anyway, which is exactly the situation the next point is about.

The lesson that generalises: build a thin internal seam early. Define one interface that your agent talks to, pick one canonical shape for messages and tool calls, and put each model family behind an adapter that translates its wire format to and from that shape. Concretely, the boundary is about this small:

# one interface the agent depends on; one adapter per model family behind it
class ModelClient(Protocol):
    async def respond(self, messages: list[Block], tools: list[Tool]) -> list[Block]: ...

# AnthropicClient, ConverseClient, ... each satisfy this and own the
# translation between their provider's wire format and the canonical Block.
def make_client(model_id: str) -> ModelClient: ...  # fail-closed: unknown id raises

Your dispatch loop only ever sees the canonical block, never a vendor payload. Make the factory fail-closed: an unrecognised model ID should raise, never silently route to a “best guess.” Fallback is only safe if the fallback path is one you’ve explicitly built and tested, not one your code stumbles into.

This is also why a framework like LangChain or LiteLLM is not a shortcut past the problem. It hands you a ready-made version of that seam across providers, which is genuinely useful, but it abstracts the wire format, not the behaviour. The per-model prompt sensitivity, the tool-calling quirks, the caching mechanics: those leak straight through any unified interface, yours or off-the-shelf.

Pitfall 2: models are not drop-in equivalents, even at equal “quality”

The second hard lesson is that two models can both be excellent and still not be interchangeable in your harness.

Concrete example: provider-specific prompt caching. The cost and latency model of an agent that re-sends a large tool-and-context preamble every turn depends heavily on prompt caching, and the way you mark cacheable spans (and even how many cache breakpoints you are allowed) is vendor-specific. Switch families and your carefully tuned caching strategy simply does not apply; your costs and latency move, sometimes sharply.

Another: model-version quirks. We found a specific model version would occasionally malform its tool calls in a way we had to detect and repair in the dispatch loop. That repair is correct for that version and meaningless for every other model. Tool-calling reliability, instruction-following under pressure, willingness to say “I don’t know” instead of fabricating: these vary enormously between models and are exactly the behaviours an agent lives or dies by.

So “best way to use a model” is real, and it is per-model: the prompt that gets the best out of one model is not the prompt that gets the best out of another, and the safety posture that one model respects, another ignores. A fallback model isn’t a spare tyre of the same size; it’s a different vehicle that happens to drive on the same roads. Treat the migration to it as a real piece of engineering, scoped and tested ahead of time, so that on the day you actually need it you are flipping a switch you have already proven.

Pitfall 3: a fallback ladder is also a cost ladder

The third consideration bites only after you have shipped: the model you fail over to has a different price, and the Region you are obliged to run it in has a different price again. On Bedrock these are two separate effects worth keeping straight. Token pricing varies widely by family, often by an order of magnitude between a frontier model and a lighter open-weight one, which is visible on the public Bedrock pricing page. Separately, Cross-Region inference itself does not add a surcharge: a request is billed at the inference profile’s published rate, which for current AWS profiles matches the on-demand rate of that profile’s primary Region. What moves the number is which profile you are obliged to use. The EU inference profile that keeps data in-region can sit above the cheapest on-demand Region for the same model. In our own cost modelling we carry roughly a 10% uplift on the EU inference-profile routes for our primary family against the equivalent US on-demand rate, and we treat that as the standing price of residency.

So a fallback ladder is also a cost ladder, and the two do not move together. Failing over to a cheaper open-weight model can save money while costing you quality; failing over to a residency-compliant route can cost more for the same model. Work both deltas out in advance, so a failover event doesn’t arrive as a billing surprise stacked on top of an incident.

Where Bedrock genuinely shines: cheap, isolated, side-by-side evaluation

Here is the flip side of all that subtlety: because every model lives behind the same Bedrock access surface, comparing them becomes an infrastructure problem you already know how to solve, not a procurement project per vendor.

We stood up a second, isolated runtime (same agent code, separate deployment, separate logs, separate metrics namespace) whose only job is to run candidate models against hard, representative tasks without touching production. A few design choices made this evaluation trustworthy, and they generalise well:

• Isolate it at the infrastructure level, not by convention. A separate runtime, image tag, log group and metrics namespace mean eval traffic can never pollute production dashboards or alerts, and a candidate model can never accidentally take a real action. Make the isolation fail-closed: if the eval deployment is missing its explicit configuration, it should refuse to deploy rather than fall back to production settings.
• Grade blind. If the evaluation environment can read the “right answer” (a human’s resolution notes, a linked root-cause record), a weaker model can look strong by quietly reading the answer key. Strip those inputs so you are measuring reasoning, not retrieval of the solution.
• Run a harness-fit probe before you blame the model. When a candidate underperforms, the natural reaction is “our prompt isn’t tuned for it.” So test that hypothesis directly: harden the prompt specifically for the candidate and re-measure. Our most valuable single finding came from this: the gap between our primary model and the alternatives was mostly model-intrinsic, not a prompt artefact. That told us the seam was worth keeping for break-glass resilience, but that switching the default wasn’t justified yet. You only learn that by measuring.

A safety note that bears repeating, because it surprised us: a “dry-run” flag that suppresses one kind of side effect doesn’t suppress all of them. In our case, suppressing the agent’s writes did not suppress its reads against live infrastructure. If a candidate model can call tools, those calls execute for real during evaluation. The durable backstop is least-privilege, read-only credentials at the boundary, not a flag in your application code. Defence in depth applies to your evaluation environment too.

Running your own model: removing the API from the critical path

Configuring fallbacks hedges against one model being unavailable. Running your own hedges against a different risk: not wanting your core workflow to depend on a third-party inference API at all, whether for sovereignty, predictable capacity, or a model fine-tuned on your own domain. The point of doing it on Bedrock is that the operational surface barely changes when the weights become yours: the same IAM controls and the same API, with Guardrails layered on where the model architecture supports them. Amazon Bedrock Custom Model Import brings supported open-weight architectures behind that surface; Amazon Bedrock Marketplace widens the catalogue; and Provisioned Throughput reserves dedicated capacity for steady, latency-sensitive load. Because the surface stays the same, a single seam can mix managed and self-hosted models on the same ladder.

The honest caveats are real but different from classic self-hosting. With Custom Model Import the serving and autoscaling stay AWS-managed (billed by Custom Model Units, with cold-start latency on an idle model), so what you take on is the cost model, the supported-architecture limits, and a quality bar an open-weight model may not clear for your task, not server ops. You only own capacity planning and scaling if you go all the way to your own Amazon SageMaker or EC2 endpoints, outside Bedrock. For most teams the right posture is a hybrid:

• Primary: a strong managed model. Your default. The one you have evaluated hardest and trust unattended.
• Fallback: a tested alternative, break-glass. Already proven through the seam on a normal day, not discovered during an outage.
• Self-hosted: for workloads where control wins. Reserved for cases where sovereignty or capacity genuinely outweighs the operational cost.

What we’d tell our past selves

• Build the seam before you need it. One internal interface, one adapter per model family, fail-closed routing. Retrofitting this under outage pressure is miserable.
• Treat “switch to the fallback” as engineering, not configuration. Prompts, caching, tool-calling quirks, and safety posture are all per-model. Prove the fallback works on a normal day.
• Default to your best model; keep the alternative warm. The point of the seam often isn’t to leave your strongest model; it’s resilience and the option to re-evaluate as the field moves.
• Make evaluation a first-class, isolated environment. Blind grading and a harness-fit probe will tell you whether your problem is the model or your prompt, saving you from both over-engineering and false economy.
• Put the real safety control at the boundary. Read-only, least-privilege credentials and Guardrails protect you regardless of which model is behind the seam, including during evaluation.

Those five are tactics. The shift underneath them is the real payoff. Amazon Bedrock did not make the model-specific subtlety disappear, and nothing will. What it changed is where the subtlety lives: behind one access surface, one security model, and one bill that we own, instead of scattered across vendor relationships we could only hope held. “Keep the agent running when a model goes dark” stopped being a procurement question and became an architecture decision.

Written at Cloudar, an AWS Premier Tier Services Partner. The lessons here come from production experience building AI-assisted operations tooling on AWS.

The post When the model goes dark: keeping your AI agent available on Amazon Bedrock appeared first on Cloudar.

The problem? For most organisations, it delivered exactly what you’d expect when you relocate an existing environment to a new address: the same bottlenecks, the same inefficiencies, the same complexity. Just with a monthly invoice attached. Fast forward to today.

The invoice as a symptom

The appeal of lift & shift is understandable. No fundamental architecture changes. Limited impact on running applications. Minimal business disruption. A fast time-to-cloud that was easy to sell internally. For a specific type of workload – end-of-life systems or legacy environments you’rewinding down in the short term – it’s still a legitimate tactic today. Especially if it lets you shut down an entire data centre. But as a migration strategy, it’s postponement, not a real solution.

On-prem systems were designed for a world of fixed capacity. Servers run continuously whether the load justifies it or not. Redundancy lives in hardware, not in design. The economic logic of cloud is precisely the opposite: you pay for what you use and scale to what you need. That partnership between IT and Finance has a name: FinOps. The discipline that ties cloud spending to actual business value. Bringing an on-prem architecture to the cloud unchanged means importing the inefficiencies of the old model into an environment that was never built for them. And paying for that continuously.

In practice, this means: virtual machines running 24/7 when the workload doesn’t require it (idle resources), storage not tiered by access frequency, and applications moving data unnecessarily between regions or availability zones, each generating data transfer costs. Without FinOpsdiscipline, no one in the organisation has a clear picture of what the cloud actually costs per application, per team, or per business unit.

The result is an invoice that runs higher than the data centre you left, with less flexibility than you expected. Not because cloud is too expensive. But because you never made the shift from ‘infrastructure in the cloud’ to ‘cloud-native architecture with cost accountability’.On-prem doesn’t fix that. It just moves the problem and adds five-to-seven-year hardware depreciation cycles to the books. The costs become slightly less visible, but you’ll feel them regardless. Especially as hardware prices keep climbing due to supply constraints.

KarmAI

For years, the architectural debt of lift & shift was a slow-burning problem. Systems ran, bills were higher than expected, but business carried on. AI has changed that. Not gradually, but abruptly.

AI workloads require infrastructure that is fundamentally different from what classic enterprise applications need. Data must be accessible easily and at scale, not locked in legacy storage structures that were once migrated one-to-one. Integration with managed services like Amazon Bedrockworks better when your environment is built on cloud-native standards. AI workloads also have a volatile cost profile: a training job or inference spike can consume more in hours than a classic application does in a month. Without FinOps mechanisms to monitor, cap, and allocate those spikes, you lose all control over your AI budget.

But that’s only the short-term pain. Because organisations that did lift & shift also migrated their data mess along with everything else. No proper classification, no clear ownership, no consistent structure. For classic applications, that was annoying but manageable. For AI, it’s fatal. You can’tfine-tune a language model on unstructured data scattered across poorly managed cloud systems. You can’t build a RAG architecture if you don’t know where your data lives or what its quality is.

Organisations that are serious about AI are being forced to revisit their cloud architecture anyway. What once looked like a technical problem has become a strategic delay… right at the moment AI is starting to make the competitive difference. And organisations that now decide to return to on-prem entirely will only widen that gap.

On-prem is capitulation

Organisations considering cloud repatriation today are asking themselves the wrong question. Because the discussion isn’t really about cloud versus on-prem. It’s about focus and resource allocation: what does your organisation actually need, and are you capable of managing it well?

Is it five to midnight for some organisations? Yes. But for those who commit to a cloud-native approach, it will become clear that cloud was the right choice all along. And that anyone thinking about repatriation today will find there’s still a great deal of cloud potential left untapped.

If the cloud hasn’t delivered on its original promises, it’s usually because the architecture never became truly cloud-native. Data that was never properly structured. FinOps discipline that was never built in: no centralised cost reporting, no shared accountability between IT and business, no culture where teams think about the cost of what they build and run. And a mindset that carried on-prem patterns into an environment where they don’t belong.

You don’t solve those problems by switching environments. You solve them by acknowledging them and addressing them. And that can absolutely be done in the cloud… if you’re willing to do it differently this time.

The post Back to on-prem? appeared first on Cloudar.

Somewhere in Europe, a tram is running right now with a bearing fault that will cause a breakdown in eleven days. The onboard sensor has been registering it for weeks. But no one caught it. Why? Because no one was looking in the right way.

Modern vehicles are loaded with sensors. Vibration, temperature, brake pressure, power consumption: every detail is measured. But a sensor is just a sense organ, not a brain. Local alarms are reactive: they tell you what has already gone wrong. The real shift happens when you stop looking at data vehicle by vehicle, and start looking across an entire fleet. When hundreds of vehicles stream their telemetry to a central cloud platform, something fundamentally different emerges: fleet-level intelligence.

AWS provides a proven foundation for exactly this. Through AWS IoT Core, sensor data from thousands of vehicles – whether trains, trams, or buses, at any scale – flows in real time to a central platform. And what you see from there changes the nature of your decisions entirely.

The platform doesn’t just see that vehicle 47 is showing an unusual vibration. It sees that every vehicle travelling a particular route today showed the same anomaly. And from that, it draws the right conclusion: the problem isn’t the train. It’s the track. That’s the difference between raising a maintenance ticket and preventing an incident.

Cost vs ROI

Traditional preventive maintenance replaced parts on fixed schedules: expensive, inefficient, and often unnecessary. Cloud-based condition monitoring inverts that logic. You maintain what needs maintaining, exactly when it needs it.

For a COO, that means higher fleet availability and fewer unplanned disruptions. For a CFO, it means maintenance costs that move with reality rather than with a calendar. For a CEO, it means a direct, structural impact on the reliability promise made to passengers.

The cloud also democratises this capability. Large national rail operators have the scale to build their own data infrastructure. Regional transit authorities typically don’t. Through AWS, even a regional bus operator gains access to enterprise-grade analytical computing power, without the capital investment in hardware or specialist data teams.

But let’s be direct: data infrastructure alone solves nothing. The most common failure we see in transport organisations is not a lack of data. It’s the opposite: an abundance of data that no one actively uses. Vast amounts of operational information gets stored and never converted into decisions. A cloud analysis that identifies an imminent fault has zero value if getting that insight to the maintenance team takes three days of bureaucracy.

This is precisely where the role of the CTO and CIO extends beyond architecture choices. Technology can only prove its value if the organisation around it is built to act on insights. Technology and organisation must evolve together.

Public trust as a strategic objective

Public transport stands at a historic inflection point. The mobility transition asks people to swap their car for a train, tram, or bus. But they’ll only do that if they trust the system. Safety and punctuality are not operational parameters. They are the pillars on which that trust rests and on which the legitimacy of your organisation as a public service provider depends.

That trust isn’t earned once. It’s built daily, or quietly eroded. Every delay that could have been prevented, every breakdown that feels avoidable, every passenger who gives up and goes back to the car: these are small fractures in a foundation that took generations to build.

The technology available to us now offers something that was unthinkable not long ago: the ability to see those fractures before they form. An organisation that understands its fleet as a learning system – one that recognises patterns, flags anomalies, and gives context to raw data – is an organisation that can make a promise to its passengers and actually keep it.

That is ultimately what this is about. Not technology as an end in itself, but what it makes possible: a public transport system so reliable that choosing the train or the bus becomes second nature. Not a conscious trade-off anymore, but a habit. And habit is the highest form of trust.

The post Tram, train, bus… the future is fleet-level intelligence appeared first on Cloudar.

A challenging process

The recognition is the result of an intensive approval process. While global players recently announced the same status, Cloudar is the first partner with a local Benelux presence to have achieved this designation. To qualify, a partner must be able to demonstrate proven revenue and a substantial Anthropic-related pipeline — a threshold that only a handful of AWS partners worldwide manage to meet.

In concrete terms, this means that Cloudar is now the only Benelux partner able to:

• Deliver enterprise seats for Anthropic
• Fully onboard customers onto the latest Anthropic models via AWS Bedrock
• Build AI solutions based on models such as Claude Code, directly on AWS infrastructure

Since the AWS policy change of October 2025, access to the latest Claude models via Bedrock requires the managing AWS partner to be officially recognised by Anthropic. Customers working with an unrecognised partner simply cannot activate these models, regardless of their own intent or budget. As a result, large organisations in the Benelux have been left out in the cold for several months. Cloudar’s recognition now puts an end to that.

Strategic advantage in a competitive market

“This partnership is no coincidence. It is the result of years of deliberate choices. From the very beginning, we committed to AWS as our sole platform and to AI as our biggest growth pillar. The Anthropic Authorized Reseller status confirms we are on the right track. For our customers, it means they can start working today with the most advanced AI models on the market, supported by a local team that guides them through the entire journey — from architecture to governance to optimisation,” says Senne Vaeyens, Co-Founder and Managing Partner at Cloudar.

As the first local Benelux partner with this recognition, Cloudar can now offer organisations in the region a complete answer to their AI challenges, within the familiar AWS environment they already use. Whether it concerns a first AI experiment or a production-ready implementation on AWS Bedrock: Benelux customers no longer need to look to international players to get there.

The post Cloudar is first local Benelux Anthropic Authorized Reseller appeared first on Cloudar.

The first one was filed on November 19, 2017. Seven years ago. The ask: let an admin change a customer’s email address in Jira Service Management. Not migrate accounts through a four-step workaround involving Atlassian ID. Just… change an email address. The kind of thing you’d expect a junior developer to ship on a Tuesday afternoon. As of today, it has 779 votes, 447 watchers, and a status of “Future Consideration.” Which, in SaaS-speak, translates roughly to: “we heard you, we filed it, please stop asking.”

The second one landed in January 2019. The request: make Confluence’s alphabetical page sorting persistent, so new pages don’t just pile up at the bottom like uninvited guests. Six years later: 600 votes, 261 watchers, status “Under Consideration.” Progress!

Now, to be fair to Atlassian, they’re not uniquely terrible. Every major SaaS vendor has a graveyard of feature requests exactly like these. Sensible, obvious, clearly wanted by thousands of paying customers. Just… never quite prioritized. Because they have a roadmap. And you’re not on it.

This is the part of the SaaS brochure they don’t show you.

The Pitch vs. The Reality

SaaS vendors are exceptionally good at one thing before you sign: making you feel like you’re about to get exactly what you need. The demos are polished. The slide decks are beautiful. The onboarding is smooth. Eighty percent of your requirements? Covered, on day one.

It’s that remaining twenty percent where things get interesting.

That twenty percent is where your actual workflows live. The edge cases. The things specific to how your organization actually operates. And when you file a support ticket asking about them, you enter a fascinating parallel universe where time moves differently. Features are “on the roadmap.” Updates will come “in a future release.” Your vote has been registered. Thank you for your feedback.

Meanwhile, you’re paying. Every month. For the product as it exists, not as it was promised.

The Numbers Are Telling

The scale of SaaS sprawl is difficult to overstate. According to BetterCloud’s annual State of SaaSOps report, the average number of SaaS applications per company peaked at 130 in 2022 and even after a wave of consolidation, still sits at over 100 today. That’s more than 100 subscriptions to manage, renew, secure, and integrate. For every single organization.

The waste embedded in that sprawl is just as striking. Gartner estimates that approximately 30% of purchased SaaS licenses go unused, what they bluntly call “toxic spend.” BetterCloud puts a price tag on it: companies report wasting an average of more than $135,000 per year on unused software licenses alone. And Gartner projects SaaS spending will continue growing at around 19% annually, reliably outpacing the budgets meant to fund it.

Do the arithmetic on a typical mid-market tool. Fifty users. €50 per user per month. That’s €30,000 per year. Every year. With annual price increases that arrive in your renewal email as a polite fait accompli. After five years, you’ve spent €150,000-plus on software you don’t own, can’t modify, and can’t easily leave and somewhere between a quarter and a third of those licenses have been sitting idle.

Something Changed

Here’s what’s different in 2026: building software got dramatically cheaper and faster. Not incrementally but by an order of magnitude.

Tools like Claude Code and Kiro represent a new category of agentic coding assistants. They don’t just suggest the next line. They can take a requirement, reason through a solution, write the code, test it, catch errors, and iterate, with minimal human supervision. What previously required a team of developers and months of work can now be done by one technically capable person in days.

This isn’t science fiction. It’s happening right now in engineering teams across the world.

And it fundamentally changes the math on one of the oldest questions in IT: should we build or buy?

What You Get When You Build Your Own

When you build a custom application -even a relatively simple internal tool- you build exactly what you need. No more, no less. You control the data model, the workflow, the integrations, and the roadmap. And critically: you decide what gets built next. Not a product manager in Sydney who’s never seen your workflows.

Running on cloud-native infrastructure like AWS means scalability, security, and availability are largely handled by the platform. The operational gap between “something you built” and “something a vendor hosts for you” has narrowed considerably. And the cost gap has flipped.

A custom-built equivalent to that €30,000/year SaaS tool, developed with AI-assisted tooling and running on serverless AWS infrastructure, might cost a fraction of that annually in operational expenses, after a one-time build investment that has also come down dramatically. More importantly: you own the asset. It does exactly what you need. And you’re not waiting seven years for someone to let you update an email address.

A Word of Honesty

Custom software isn’t free of responsibility. Someone has to maintain it, secure it, and evolve it. The SaaS argument, that someone else handles the infrastructure, the uptime and the patches, is not without merit. And for commodity functions like email, video conferencing, or payroll, that argument still wins. These are solved problems. Building your own would be an expensive act of reinvention.

But pairing custom-built applications with managed cloud services gives you the operational coverage of SaaS without the product dependency. You get uptime. You get security. You get to decide what comes next.

So What Should You Actually Do?

Not everything should be custom-built. That would be its own kind of madness.

The question worth asking is simpler: where are your core differentiating workflows? The processes that encode years of operational knowledge. The edge cases that keep showing up in feature request threads, yours and everyone else’s, year after year, status unchanged.

Those are exactly the features that will never quite make it onto a SaaS vendor’s roadmap.

The question is no longer “can we afford to build?” The question, in 2026, is whether you can afford to keep waiting.

JSDCLOUD-5746 would like a word.

The post Dear SaaS Vendor, We’ve Been Waiting Since 2017. appeared first on Cloudar.

Creating and managing secrets is perhaps as old as humans interacting with each other. Yet despite their critical importance, secrets remain one of the most vulnerable aspects of AWS infrastructure today. In our practice as Dev(Sec)Ops Engineers, we see these challenges daily accross our client environments.

To start, the question needs to be asked: “What exactly is a secret?”  In AWS and Cloud environments in general, a secret is anything used to access systems and/or data. These digital keys unlock potentially critical systems and must be protected from unauthorized access at all costs.

As Dev(Sec)Ops Engineers working in AWS, you are responsible for managing an extensive arry of secrets:

• RDS database usernames and passwords
• API keys for AWS services and third-party integrations
• CodeDeploy and deployment system credentials
• KMS encryption keys for data protection
• Private keys for SSL/TLS certificates and secure communications
• EC2 SSH keys and key pairs
• IAM user credentials for developers, QA, and operations teams
• Application service accounts and roles
• Any username/password combinations
• Sensitive configuration data that could aid attackers

The challenge is not just the volume … it’s the complexity of securing these secrets across multiple AWS accounts and other environments while maintaining operational efficiency.

The Hidden Dangers: How Secrets Get Compromised in AWS

 The Configuration File Trap

Storing secrets in configuration files is convenient but dangerous, especially in AWS environments where these files often end up in S3 buckets or EC2 instances. This risk was nicely illustrated in the RSA conference presentation “Red Team vs Blue Team on AWS” by Kolby Allen and Teri Radichel: Link to video

In the video Kolby deployed AWS resources using sample code found easily on the Internet.  Teri, simulating an attacker, conducted a penetration test and discovered a Lambda function’s configuration file stored in a S3 bucket (for convenience) complete with REDS database credentials in plain tekst.

If you must use configuration files in AWS:

• Maintain separate files for development, QA, and production AWS accounts
• NEVER commit these files to source control systems (GitHub, GitLab, Bitbucket)
• Use S3 bucket policies and encryption, but understand this is still suboptimal
• Consider AWS Systems Manager Parameter Store as a minimum improvement

This vulnerability is so important that it’s formally recognized in the MITRE attack framework:  [CWE-200: Exposure of Sensitive Information] (link)

Public Source Control: A Goldmine for AWS Account Takeovers

The most devastating secrets management failures occur when AWS credentials are committed to public repositories. The scale of this problem is staggering.

The people at TruffleHog constructed a live scanner for this:

https://forager.trufflesecurity.com/explore

This scanner continuously detects AWS access keys, secrets keys and other credentials in public Github commits, revealing the massive and constant stream of exposed AWS and other secrets.

This has a widespread impact: TruffleHog researchers discovered approximately 4,500 secrets among the top 1 million websites, many of which were AWS credentials, as detailed in their comprehensive analysis:

https://trufflesecurity.com/blog/4500-of-the-top-1-million-websites-leaked-source-code-secrets

Leaking these secrets can have far reaching financial consequences:

A Reddit user faced a 26.000 $ bill after IAM was compromised to execute crypto miners:

https://www.reddit.com/r/aws/comments/17p3v1e/account_got_hacked_and_get_26000k_bill/

A Developper was billed 14.000 $ on AWS following similar exposure:

https://dev.to/juanmanuelramallo/i-was-billed-for-14k-usd-on-amazon-web-services-17fn

AWS responded on these massive bills and is now actively scanning GitHub respositories through their AWS Credentials Exposed program and automatically disables discovery IAM access keys, but the frequency of these incidents remains alarmingly high as shown by the Trufflehog data.

Internal Systems: The False Security Blanket in AWS

Private repositories and internal systems create a dangerous illusion of security, even within AWS environments. The 2020 Twitter breach perfectly illustrates this vulnerability:

Attackers breached the perimeter, accessed internal Slack channels where developers had stored AWS credentials and other secrets, and used these to compromise infrastructure in a widely publicized incident:

https://www.zdnet.com/article/twitter-says-hackers-accessed-dms-for-36-users-in-last-weeks-hack

Secrets embedded in code even in fully private AWS services, proliferate across AWS services and do appear in:

• CloudWatch logs from Lambda functions and EC2 instances
• S3 bucket access logs
• CloudTrail event data
• Application Load Balancer logs
• Systems Manager Session Manager history

As such they do create additional attack vectors within your AWS environment.

Runtime Exposure: Secrets in AWS Production Workloads

Running AWS applications create numerous opportunities for secret exposure:

• Configuration files within EC2 instances or container images
• Environment variables visible in ECS task definitions or Lambda configurations
• Memory dumps from EC2 instances containing sensitive data
• Application caches in ElastiCache storing credentials
• CloudWatch logs revealing secrets in error messages
• EC2 instance metadata (IMDSv1) exposing IAM credentials
• Unencrypted S3 bucket metadata and tags
• AWS CloudShell command history
• Bash history on EC2 instances
• Container image layers in ECR with embedded secrets

This non-exhaustive list demonstrates how secrets can leak from multiple AWS vectors without proper handling.

What does AWS offer to fight this battle ? AWS Native Secrets Management

Adopt Just-in-Time Secret Retrieval with AWS Services

Core Principle: Store secrets in AWS-native management systems and retrieve them only when needed.

Instead of embedding secrets in configuration files, implement this AWS-secure workflow:

1. Store secrets in AWS Secrets Manager or Systems Manager Parameter Store
2. Retrieve secrets programmatically using AWS SDKs at runtime
3. Use IAM roles and policies for access control
4. Clear secrets from memory when no longer needed

AWS Access Control: Implement least-privilege IAM principles

• Developers cannot access production secrets via IAM policies
• Applications use IAM roles to retrieve only their required secrets
• Cross-account access is controlled via resource-based policies

AWS-Native Secrets Management Solutions

• AWS Systems Manager Parameter Store
• AWS Secrets Manager (Recommended for Production)

As an MSSP, we recommend AWS Secrets Manager as the gold standard for AWS environments.

Why Secrets Manager over Parameter Store?

• Enhanced security controls with fine-grained IAM integration
• Automatic secret rotation for RDS, DocumentDB, and Redshift
• Cross-account access capabilities essential for multi-account AWS strategies
• Comprehensive audit trails via CloudTrail integration
• No CloudFormation exposure risks unlike Parameter Store
• Encryption at rest using AWS KMS by default

As such we recommend the AWS Secrets Manager for Production workloads, the Encrypted Parameter Store can be used as a configuration store to fetch certain parameters.

How to use this in code ?

Python with boto3:

import boto3
import json

# Using IAM role-based authentication (recommended)
client = boto3.client('secretsmanager', region_name='us-east-1')

try:
    secret_response = client.get_secret_value(SecretId='prod/rds/mysql-credentials')
    secret_dict = json.loads(secret_response['SecretString'])
    
    # Use the secret
    db_username = secret_dict['username']
    db_password = secret_dict['password']
    
except ClientError as e:
    # Handle AWS-specific errors
    if e.response['Error']['Code'] == 'DecryptionFailureException':
        # Secrets Manager can't decrypt the protected secret text using the provided KMS key
        raise e
    elif e.response['Error']['Code'] == 'InternalServiceErrorException':
        # An error occurred on the server side
        raise e
    elif e.response['Error']['Code'] == 'InvalidParameterException':
        # Invalid parameter value
        raise e
    elif e.response['Error']['Code'] == 'InvalidRequestException':
        # Parameter value is not valid for the current state of the resource
        raise e
    elif e.response['Error']['Code'] == 'ResourceNotFoundException':
        # Can't find the resource that you asked for
        raise e

Via Infrastructure as Code:

• Cloudformation:
  • https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-resource-secretsmanager-secret.html
• Terraform:
  • https://docs.aws.amazon.com/prescriptive-guidance/latest/secure-sensitive-data-secrets-manager-terraform/using-secrets-manager-and-terraform.html
• AWS CDK:
  • https://docs.aws.amazon.com/secretsmanager/latest/userguide/cdk.html
• Pulumi:
  • https://www.pulumi.com/registry/packages/aws/api-docs/secretsmanager/secret/

AWS Key Management Service (KMS)

AWS KMS provides dedicated encryption key management, solving the “where to store the encryption key” problem for AWS environments. KMS integrates seamlessly with Secrets Manager and most AWS services.

KMS Best Practices for Secrets:

• Use customer-managed KMS keys for production secrets
• Implement key rotation policies
• Use separate KMS keys per environment/account
• Leverage KMS key policies for fine-grained access control

AWS Systems Manager Parameter Store

While we recommend Secrets Manager for sensitive data, **Parameter Store** works well for:

• Non-sensitive configuration data
• Cost-conscious environments (free tier available)
• Simple use cases without rotation requirements

Multi-Account AWS Strategies

For AWS Organizations with multiple accounts (our recommended approach), consider:

Cross-Account Secrets Access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::PROD-ACCOUNT-ID:role/ApplicationRole"
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "secretsmanager:ResourceTag/Environment": "production"
        }
      }
    }
  ]
}

Multi-Region Considerations:

• Replicate critical secrets across AWS regions
• Use AWS Secrets Manager automatic replication
• Consider data residency requirements

AWS Implementation Best Practices from Our MSSP Experience

Security Architecture Considerations

• Design secure AWS deployment pipelines** using CodePipeline, CodeBuild, Github Actions with ci-cd plugins like trufflehog : https://undercodetesting.com/how-to-hunt-for-sensitive-credentials-using-trufflehog
• Implement comprehensive IAM access management with least-privilege principles
• Establish governance policies using AWS Config and AWS Organizations SCPs/RCPs
• Plan for secret rotation using AWS Secrets Manager automation
• Monitor and audit secret access via CloudTrail and CloudWatch

Operational Excellence in AWS

• Automate secret provisioning using AWS Lambda and CloudFormation/Terraform/…
• Implement emergency access procedures via AWS SSO and break-glass roles
• Establish incident response for compromised secrets using AWS Security Hub or other CSPM/CNAPP
• Regular security assessments using AWS Inspector and third-party tools
• Cost optimization by right-sizing Secrets Manager usage vs. Parameter Store

AWS-Specific Monitoring and Alerting

Set up CloudWatch alarms for:

• Unusual Secrets Manager API calls
• Failed secret retrievals
• Cross-account secret access
• KMS key usage anomalies

Conclusion

Effective AWS secrets management is not just about choosing the right AWS service it’s about implementing a comprehensive security strategy that leverages AWS-native capabilities while addressing the entire lifecycle of sensitive data. The examples and breaches discussed here represent real financial and reputational risks that AWS customers face daily.

Key AWS Takeaways:

1. Never store secrets in configuration files, S3 buckets, or source control
2. Use AWS Secrets Manager for production secrets management
3. Implement just-in-time secret retrieval with AWS SDKs
4. Apply least-privilege IAM policies and roles
5. Leverage AWS KMS for encryption key management
6. Plan for multi-account and multi-region secret strategies

The post AWS Secrets Management: Protecting Your Digital Keys in the Cloud appeared first on Cloudar.