Interestingly, Roles Anywhere does not completely solve the problem of not needing to create long-lived credentials, instead it moves (and reduces) the problem from credential management (rotation) to certificate management (distribution).

Setting up a Certificate Authority (CA) that can create certificates can be done with AWS Private Certificate Authority, and there are some options out there to manage distribution (usually as part of a bigger Device Management solution), but wouldn’t it be nice if the government of Belgium would do that for us?

The Belgian eID

In Belgium every citizens gets an identity card. The government makes sure that these get distributed and renewed, and since 2005 they contain a chip that holds (among other things) a certificate that can be used for digital authentication. If we can tie this to Roles Anywhere, we don’t have to worry about certificate management ourselves.

This eID is the size of a bankcard and the chip can be interacted with using off the shelf smart card readers. Most information can be read without any additional authentication – if you have the correct middleware, but to sign something you need to enter a PIN code.

The different certificates

Every eID holds an “authentication certificate” that is tied to the owner of the eID and has been signed by the “Citizens CA”.  The Citizens CA’s certificate is in turn signed by the Belgium Root CA . The Root certificate is self-signed. If we trust the Belgium Root CA, we will be able to validate any authentication certificate. More information about this setup can be found at https://repository.eidpki.belgium.be/#/home

The authentication certificate has two pieces of information that are going to be useful to us:

• The Issuer, showing us that the certificate is indeed issued by the Citizen CA.
• The Subject, containing the name and National Number (a number that uniquely points to one person) of the owner.

AWS IAM Roles Anywhere

Roles Anywhere allows you to use a certificate and its private key to get temporary credentials. You can think of it as an AssumeRole call, but instead of starting from an Access Key, we start from a certificate.

There are a few components we need:

• Configuration in our AWS Account, to indicate which CA we trust, and which roles can be assumed by different Issuer and/or Subjects. This also includes the creation of IAM Roles.
• Software on our computer to interact with the Card Reader so we can prove we have access to the eID and know its PIN.

For our use case we are going to trust the Belgium Root CA, and create a Role that can be assumed with any certificate issued by the Citizen CA. We will than limit the permissions of that IAM Role, so it can only write to a unique prefix in S3.

We can do that because every component of the Subject in our certificate gets mapped to a principal tag on the role session. E.g. if the subject is “C=BE, SN=Doe, GN=John, serialNumber=70010112345, CN=John Doe (Authentication)”, we have the following principal tags, and we can use principal tags in any policy.

• x509Subject/C: BE
• x509Subject/SN: Doe
• x509Subject/GN: John
• x509Subject/serialNumber: 70010112345
• x509Subject/CN: John Doe (Authentication)

Putting it all together

In our AWS Account

We will create the following resources in our AWS Account:

• An S3 Bucket
• A trust anchor, configured to trust the Belgium Root CA certificate that is publicly available via https://crt.eidpki.belgium.be/eid/brca6.crt.  This way we can use any certificate created by the Root CA of any of its subordinate CAs (like the Citizen CA)
• An IAM Role, where we configure the trust policy to only allow usage of the Role if:
  • The principal is the Roles Anywhere service (rolesanywhere.amazonaws.com)
  • The Issuer is Citizen CA (aws:PrincipalTag/x509Issuer/CN must equal “Citizen CA”)
  • The source is our trust anchor (aws:SourceArn equals the ARN of the Root CA trust anchor)
• An IAM Policy on this role that allows PutObject acces to all objects that start with the unique National Number (aws:PrincipalTag/x509Subject/serialNumber)
• a (Roles Anywhere) profile that ties the trust anchor to the Role. We could configure more settings here, like a mapping of fields from the certificate or a session policy, but don’t need to.

Of course we do not have to do this by hand, we create a CloudFormation template for this. It can be found in our sample repository: https://github.com/WeAreCloudar/cloudformation-samples/tree/main/templates/roles-anywhere-eid

On our machine

On our machine, we will need – besides the eID and its PIN (at least on macOS, where I tested this):

• a SmartCard Reader
• the Roles Anywhere Credential Helper (downloadable from the AWS Website)
• The AWS CLI (or any other tool we want to use with our AWS Credentials)

We also need a few pieces of information:

• The ARNs of the trust anchor, profile and role we created
• Our own National Number and the name of the bucket (to test our policy)

Ideally we’d be able to configure the credential helper to read any eID we insert, and this is possible if you install the Belgian eID Middleware using the following command.

aws_signing_helper credential-process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--certificate "pkcs11:model=Belgium%20eID;object=Authentication" \
--pkcs11-lib '/Library/Belgium Identity Card/Pkcs11/beid-pkcs11.bundle/Contents/MacOS/libbeidpkcs11.dylib

However because of an issue with the helper, this only works on macOS if you build the helper yourself from source. Since we do not want to do that, we can plug in our eID and find and identifier using “aws_signing_helper read-certificate-data”. You should get something like

Matching identities
1) bf013dd57aa34f8f9d4e22ab89dd8eb2 "SERIALNUMBER=70010112345,CN=John Doe (Authentication),C=BE,..."
2) 786f7572198a4a8a82ee7ef3f205d7be "SERIALNUMBER=70010112345,CN=John Doe (Signature),C=BE,..."


Copying everything between quotes, allows us to create a new file, selector.json that looks like this:


[
{
"Key": "x509Subject",
"Value": "SERIALNUMBER=70010112345,CN=John Doe (Authentication),C=BE,..."
}
]

and use that to select our specific certificate:

aws_signing_helper credential-process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--cert-selector file://selector.json

Our smart card reader will now prompt us for a PIN. If we enter the right one, we get credentials as output:

{"Version":1,"AccessKeyId":"ASIA...","SecretAccessKey":"COA...","SessionToken":"IQoJ...","Expiration":"2025-03-28T14:16:07Z"}


We can put this command in our ~/.aws/config file, so the AWS CLI will execute it for us, but for testing purposes it’s slightly easier to use the “serve” option.  The AWS Documentation has a longer explanation about how you can use the credential-process: https://docs.aws.amazon.com/rolesanywhere/latest/userguide/credential-helper.html#credential-helper-examples

Testing with s3

In one terminal we can run a credential / metadata server that can be used by the AWS CLI:

~$ aws_signing_helper serve -process \
 --trust-anchor-arn $trust_anchor_arn \
--profile-arn $profile_arn \
--role-arn $role_arn \
--cert-selector file://selector.json


This gives us
2025/03/28 14:28:12 Local server started on port: 9911
2025/03/28 14:28:12 Make it available to the sdk by running:
2025/03/28 14:28:12 export AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:9911/

We can than use a second terminal to run:

AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:9911/ aws s3 cp example_file s3://$bucket/serialNumber/70010112345/example_file

and get our file in s3:

upload: example_file to s3://.../serialNumber/70010112345/example_file

Conclusion

This experiment shows that being able to use an external device withIAM Roles, is a very powerful feature, because it can remove a lot of tasks related to certificate and credential management. Being able to do this with something I had in my backpocket made it extra cool.

However, the Belgian eID is probably not the best way to go about this in a real scenario:

• I just gave everyone living in Belgium access to my S3 Bucket. I could use a more scoped-down trust policy, but I would eventually run into limitations of trust-policy size.
• I also didn’t tackle Certificate Revocation Lists (CRL). Every time someone loses their wallet, I would need to invalidate certain certificates – this also runs into size limitations
• I already have a good way to give users access to AWS Accounts, using Identity Center or Cognito.
• The pkcs11 support on macOS seems limited, and I would have to install the middleware everywhere,

This does not mean that Roles Anywhere is not a good solution for this kind of cases:

• If you are already putting certificates on machines (e.g using an MDM solution), AWS IAM Roles Anywhere is a great fit
• Instead of relying on the Belgian Government you can buy an hardware key like a yubikey and use it as your certificate source. This is as secure as using the eID (only the machine with the key plugged in will be able to request credentials), and you would be limiting it to the exact serial number(s) in your trust policy anyway – making CRLs mostly moot.

The post Sign in with your eID: Using AWS IAM Roles Anywhere with a SmartCard Reader appeared first on Cloudar.

You can find them below, with a short explanation of why I think you should consider attending them.

Breakout sessions

SEC209 | Modernize authorization: Lessons from cryptography and authentication

Speakers: Eric Brandwine, Neha Rungta
Hurry level: There is still space.

You might have heard the phrase “there is no compressions algorithm for experience” and with a combined 23 years at AWS/Amazon between Eric Brandwine and Neha Rungta, this is absolutely true here. I have no doubt that they’ll be able to talk about some of my favorite topics (cryptography, authentication, authorization, the Cedar language, …) in a way that’s useful and understandable.

SEC236 | The AWS data-driven perspective on threat landscape trends

Speaker: Paul Bodmer
Hurry level: You’ll probably have to sit in the front row. There will be a recording

The description mentions “data-driven”, “real-world examples”, and “unique perspective”. AWS has been recently more open about the kind of threads they see, and this session will hopefully follow that trend.

SVS323 | I didn’t know Amazon API Gateway did that

Speakers: Eric Johnson
Hurry level: You’ll probably have to sit in the front row. There will be a recording

I have never regretted seeing a talk by Eric Johnson, and I love this title. Finding interesting ways to use AWS services is very helpful, and allows you to make more informed trade-offs when designing an application.

DOP209 | Governance and security with infrastructure as code

Speakers: Damian Silbergleith Cunniff (GoDaddy), Eric Beard, Kevin DeJong
Hurry level: You’ll probably have to sit in the front row.

I like the way cfn-guard approaches governance a lot, and it’s integrated in more services than you might expect. I also believe that having guardrails is operationally a better choice than building your own company-specific constructs, so I’m happy that they’ll also talk about the CDK.

STG315 | Amazon S3 security and access control best practices

Speakers: Meg Rose, Becky Weiss
Hurry level: You’ll probably have to sit in the front row. There will be a recording

Meg Rose and Becky Weiss gave one of my favorite sessions of re:Invent 2021 (STG315: Deep dive on Amazon S3 security and access management), and this year they’re back for another joint session. This will be a great way to get to know all the S3 security features, and how they work together.

Chalk Talks

SEC316-R1 | All things in life are temporary: An IAM credential journey [REPEAT]

Speakers: Meg Peddada, Alex Waddell
Hurry level: There is still space.

I saw a great chalk talk at re:Inforce about how IAM and STS work. In the meantime, more official documentation about concepts discussed during that talk, like “forward access sessions (FAS)”, has appeared. Getting an end-to-end look at what happens when you use an IAM credential will always be interesting, and helpful when writing policies.

NET316-R1 | Networking ask-me-anything talk [REPEAT]

Speakers: Jamie Wenzel, Andrew Gray
Hurry level: There is still space.

Doing an “ask-me-anything” is always a bit of a risk, but if we bring interesting questions, we should get interesting answers. Networking also lends itself very well to whiteboarding, so I expect a very interactive session, which is where chalk talks shine.

SUP201 | Get the most out of AWS Support to achieve your business outcomes

Speakers: Peter Dachnowicz, Donald Quindardo
Hurry level: Don’t wait too long.

Knowing how to effectively engage AWS Support is one of the “secrets” that can make you solve problems a lot faster. The topics listed in the description sound very interesting, but I would also use this opportunity to ask things like “What is the fastest way to get a quota increase?”, “What’s the smartest way to engage a Technical Account Manager?”, and “What kind of artifacts should I include when I open a support case?”.

SEC217 | Explore the AWS sovereign-by-design approach

Speakers: Alex Meek-Holmes, Kathy Liu
Hurry-level: Don’t wait too long.

By the time this session happens, it will be a little over a month since AWS announced the “AWS European Sovereign Cloud”. There are a few details in the announcement post that give us a little insight into how it will work. Hearing things directly from AWS-people, and being able to ask questions should teach us even more about what is coming.

SVS203-R1 | Thinking serverless [REPEAT]

Speakers: James Beswick
Hurry-Level: Don’t wait too long.

Framing serverless as an approach to a business problem is super helpful, and this one has all the things you want in a serverless-related session: different services, customer focus, distributed systems, event-based architecture, …

This blog has been updated:

• Monday 6 November 2023: SVS323, DOP209, and STG315 have no more reservations available.
• Wednesday 8 November 2023: crossed out SEC236, un-crossed DOP209

The post There is still space available: Ben’s recommendations for re:Invent 2023 appeared first on Cloudar.

The findings

When you configure an AWS service to use an IAM role on your behalf, you need to have the iam:PassRole permission in addition to the permissions required to configure and/or use the service.

Due to the bug we reported, AWS Directory Service didn’t verify the presence of those permissions when assigning users or groups to an existing IAM role. This allowed users with the  ds:EnableRoleAccess permission to assign users to any role in their account that has a trust relationship with AWS Directory service.

Additionally, we found some actions that were not logged to CloudTrail and a validation that’s only done client-side

The discovery

Cloudar is an AWS Managed Service Provider Partner, and as such, we work together with our clients to configure their environments. One of our clients alerted us of a strange behavior in the AWS Console for AWS Directory Service, which prompted us to take a deeper look behind the scenes of how that service works.

Step 1: Missing CloudTrail Data

We had configured AWS Directory Service for AWS Management Console access for our client, and it was working properly for them. However, when looking in the console, the feature appeared to be disabled.

Usually, we would investigate CloudTrail logs to see what disabled this, but at the time, we didn’t see an entry in CloudTrail that showed the enabling or disabling of access. When we tested what happened when changing the status ourselves, we also did not see those actions recorded in CloudTrail.

We’ve reported this to AWS, and enabling or disabling console access does now create events in CloudTrail (note: **OMITTED** is part of the CloudTrail event):

{
[…]
    "eventSource": "ds.amazonaws.com",
    "requestParameters": {
        "directoryId": "d-123456789a",
        "applicationId": "***OMITTED***",
        "serviceAccountUserName": "***OMITTED***",
        "serviceAccountPassword": "HIDDEN_DUE_TO_SECURITY_REASONS"
    },
    "responseElements": null,
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "sessionCredentialFromConsole": "true"
}

{
[…]
    "eventSource": "ds.amazonaws.com",
    "eventName": "UnauthorizeApplication",
    "requestParameters": {
        "directoryId": "d-123456789a",
        "applicationId": "***OMITTED***"
    },
    "responseElements": null,
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "sessionCredentialFromConsole": "true"
}

Step 2: Looking at what the console does

Instead of looking through CloudTrail logs, we opened the development console in our browser. There we saw that it was sending requests to an undocumented API when enabling console access. There is no public API for enabling console access or assigning user to roles within Directory Service, so this wasn’t unexpected.

What was unexpected was that when enabling or disabling console access, there were two calls to the internal API. One to (un)authorize an application called “AWSManagementConsole” and one to (un)authorize an application called “Directory Service console”.

Assuming there was a consistency problem, we investigated what happened if we only unauthorized one of them by sending requests to the undocumented API from the command line.  When we unauthorized the “AWSManagementConsole” and authorized “Directory Service console”, we were able to reproduce our customer’s issue where the console would show “disabled”, but the sign-in would still work.

By the time we sent our report to AWS, the double API call didn’t happen anymore (only “AWSManagementConsole” is used today), so this seemed to have been a temporary issue. We still reported this behavior, indicating that we could no longer reproduce.

Step 3: Finding a race condition

Since this wasn’t an easy bug to uncover, we had to read a lot of documentation to deeply understand what Directory Service was doing. While doing that, we noted an interesting sentence: “If any IAM roles have been assigned to users or groups in the directory, the Disable button may be unavailable.”

The word “may” seemed very ambiguous, especially after finding an issue where a network disconnection could lead to an unexpected state. We looked a bit further and realized that the expected behavior is that you can’t disable the integration if you still have users and/or groups assigned to roles.

However, that check is done completely in the browser, and appears to be based on the state when the tab was loaded. This means you can disable the console access while still having users and/or groups assigned. You could do this by calling the same (undocumented) API that the console uses, or in the browser by 1) opening the console while there are no assignments 2) using a second tab to assign users and/or groups to a role 3) using the first tab to disable console access.

We’ve reported this issue to AWS, but don’t consider it a vulnerability as the assignments are not usable if the Console access is disabled.

Step 4: Privilege escalation

Having discovered two APIs that did some part of their logic client-side, and having experimented with the role-assignment, we looked at other validations that should happen when you assign a Role to a user in a directory.  To assign a specific role to a user or group:

• you should have ds:EnableRoleAccess permission for the Directory Service domain;
• the role you want to pass should have a trust policy that allows use in AWS Directory Service;
• the role should be in the same AWS account;
• you should have iam:PassRole permissions on the role.

When we tested these checks, we noticed that we could assign a role (within the same account), even if we didn’t have the right iam:PassRole permission. In certain setups, this could mean privilege escalation.

• Imagine you have two people, Alice and Bob, managing a domain, and each of them is responsible for assigning users to specific IAM roles. Alice assigns people to Role A and Bob assigns people to Role B. They could abuse the missing iam:PassRole check to assign people to a role they are not responsible for. For example, Alice assigns herself (or someone else) to Role B.
• Another possible scenario would be when a user has the required permissions to create and manage a domain in AWS Directory Service, but no IAM permissions (e.g., with the PowerUserAccess managed policy). If there is already a role that has a trust relationship with AWS Directory Service, that user could create their own domain and assume that existing role.

This issue has been completely fixed.

Recommended actions

Taking advantage of this vulnerability required a specific configuration and a high level of permissions within Directory Service. AWS customers who were relying on iam:PassRole permissions for access control before 2023-05-12 should review their directory for misconfiguration and their CloudTrail logs for unexpected AssumeRole calls.

Timeline

2023-04-07: Issue found and reported to AWS.

2023-04-07: Investigation started by AWS.

2023-04-13: Issue confirmed by AWS, and development of a fix in progress.

2023-04-24: Development finished and deployment of the fix in progress.

2023-05-12: Fix deployed globally.

Acknowledgements

We would like to thank the AWS teams involved, especially the Security team for their open communication and helpful responses. I’d also like to thank my coworkers for diving deep when they see unexpected things in the AWS console.

Reach out to us

I talked on June 12 at fwd:cloudsec about how this journey showed the universal benefits of security features, and you can watch that talk on youtube.

The post Spotted: How we discovered Privilege Escalation, missing CloudTrail data and a race condition in AWS Directory Service appeared first on Cloudar.

Sometimes we run into questions that seem simple, but that have a very complex answer. In this blog post we will explain why “Can I migrate between two Amazon Cognito User Pools” is one of those question, and why it usually leads to a new set of questions, like:

• Where do your users exist (do they federate from a different Identity Provider (IdP), or are they in the User Pool?)
• Are you using MFA in Cognito
• Which configuration settings can you change?
• Are you okay with a migration process in the application?
• Will all users sign in during a certain timeframe?
• Do you want to keep your endpoints and identifiers, or are new endpoints okay?

Lets look at why we are asking those questions, and which Cognito components are tricky to deal with.

Configuration

The configuration of a User Pool can be described by the API, or managed in an Infrastructure as Code tool. This makes it easy to recreate with the same settings. However, besides the new endpoints and ids (see below), you will have to update the federation settings (of you are using sign in through a third party).

Eg. If you are federating users from Active Directory (AD) using SAML, you will have to set up a new integration on the AD side as well, so that user can sign in via AD/SAML. This is the same process as setting up this IdP for the first time.

Endpoints and Ids

If you create a new user pool, it will get a new id. This means that the following things will change. Usually this is not a problem, as these are expected changes (if you migrate to a different product, similar things will change).

• The User Pool Id
• The App Client Ids and Secrets (you also create new App Clients)
• The domain used for the hosted UI. Technically you can move it, but this will include downtime. Using a new domain will always be easier (and give you more control about step-by-step updates and rollbacks)
• The User Pool Id and App Client Id are also part of the Tokens that Cognito returns. If you’re verifying them in your application (which you should), or are using a managed integration between an AWS service and Cognito (eg. Api Gateway), you will have to update these too.

Users and Groups

This is the difficult part: while there are APIs to list/describe users and groups, there is no way to export the password (hash) or the MFA (seed) of a user. This means that in the worst case scenario you have to disable MFA and all users will have to reset their password before they can log in again. There are a few solutions, but none of them are ideal.

Problem: You are using MFA in Cognito

This is the worst case scenario for a migration. There is no way to export or set MFA seeds in Cognito, and usually disabling MFA for everyone is not going to be an option

Problem: You are using the sub attribute in your application

The sub attribute is an unique identifier for every user, and migrating users will create a new identifier. To work around this, you could use a different (custom) attribute to safe an identifier in, and update your application to use that. Filling that attribute comes with its own complexities, though.

Option 1: You are using federation / sign in through a third party

You’re in luck! Since the user and their password exist in the other system, you can setup a new federation and users will still be able to sign in (but  make sure you’ve read the “Configuration” section above).

If you want to go this route to make migration (and failover) easier, there are third party IdPs that you can use via SAML/OIDC. Although in some cases it might make more sense to integrate directly with them (either in Cognito Identity Pools or in the application) instead of putting User Pools in between. One reason to keep Cognito, is that you can create multiple App Clients (and thus applications) on one pool (and this can be done with the IaC tool you are already using). Whereas if you integrate directly, you have to configure everything in the external service (and depending on the protocol you are using, this can be hard to debug)

If you are also using groups, you still have to export and recreate those. Take a look at the AWS Solution mentioned in option 2,  and its limitations for an example of doing that.

Option 2: You don’t mind resetting all passwords.

This is also doable, you can export all users, import them in the new pool and make them set a new password to sign in (if this is a limited set of internal users, you could also set a password for them, but you’d want them to pick their own either way)

There is even an AWS solution (read: Open Source software that you have to manage yourself) that can be used to do the export once or on a schedule (eg. to create backups). Make sure to also check the limitations.

Option 3: Migrating on first sign-in

You can add a Lambda Function to the new User Pool (there is example code, but you’d have to manage this yourself). This function will get the username and password, and can check the original pool and write the existing password to the new pool if the password is correct. Docs:

This has some downsides:

• The old User Pool needs to stay around until the migration is complete (you pay for active users, so this does not have to be a cost problem)
• This can take a long time, and potential never finish (if you have users that don’t sign in regularly)

Conclusion

The post Options for migrating between Amazon Cognito User Pools appeared first on Cloudar.

Indirection

With Terraform […] no intermediary sits between you and the service you’re controlling. Want an RDS instance? Terraform will make calls directly to the RDS API.

First of all, both Terraform and CloudFormation provide some level of indirection. CloudFormation converts a YAML or JSON template to a set of API calls. Terraform does the same, starting from an HCL template.

Having that transformation in the cloud can make it harder to see the underlying API responses, but that’s not because of the conversion between template and API. The same APIs are still getting called, and they will raise the same errors when there is something wrong with the requests. If you look in CloudTrail, you will see these requests together with the entire error message.

The way CloudFormation implements that translation layer (for newer resources) can add a second step after an API call fails. However, it also allows for standardized errors in other cases, like “Resource Already Exists” or “Only configurable when creating a resource”.

The prerequisite steps you have to take to use CloudFormation [StackSets] across multiple accounts also must be taken just to have CloudFormation spin up resources in multiple regions within the same account.

I’d argue that StackSets (and Service Catalog) are benefits of using CloudFormation. However, if they are too much overhead for a project, you can still deploy to multiple regions by adding a –region parameter to your deployment command. Although, by not using StackSets, you’d lose some convenience features, like being able to update multiple regions in parallel easily.

Logic

Terraform offers a rich set of data sources and transforming data with Terraform is a breeze.

I agree that CloudFormation could use more intrinsic functions, but that does not mean it’s severely lacking in functionality. In many cases, you can replace the missing functions by being stricter about the acceptable inputs.

Speed

AWS CloudFormation is S-L-O-W.

CloudFormation can seem slow because it tries very hard not to get into a state where your infrastructure is broken. Both CloudFormation and Terraform try to execute as much in parallel as possible (keeping dependencies between resources in mind). However, CloudFormation will not start deleting resources until every creation and update has been completed. On top of that, CloudFormation will validate that all actions (deletes included) finish before it considers the deployment complete.

Sync vs Async

Wrapping CI/CD tooling around AWS CloudFormation is […] a difficult matter of polling stack updates until the right stack state bubbles up from an aws cloudformation CLI query.

Here Greg argues that Terraform is better because polling for changes can be challenging to do. However, if you’re okay with executing a command-line tool (like Terraform), you can AWS CloudFormation deploy, and the AWS CLI will do all that for you.

Even without the CLI, running async comes with some significant advantages:

• You don’t have to worry about the build tool (or your laptop) crashing or losing internet connectivity
• Everyone with access to the AWS account can see when an update is in progress and poll independently for changes
• You don’t need a process that keeps running during the change. The deployment will keep running until it’s done.

Running 100% as a managed service also includes the following advantages:

• You don’t need to execute any code on your machine
• You don’t need to worry about different tool versions and upgrades between them
• You don’t need an access key with full admin permissions
• You get state management included by default (instead of having to configure it yourself).

That safer approach has the potential of doubling the time it takes to execute a change (assuming that a create/update takes as long as a delete), but having automatic rollbacks is frequently worth the extra time.

Portability

If you learn AWS CloudFormation, then guess what: you can’t take your skills with you. If you put forth the effort to learn Terraform, then you can take your skills to any other cloud.

While the workflow might stay the same, the resources that you create are going to be vastly different between clouds. The primary skill that you should learn (independently from the tool) is how AWS works. This will make your skills transferable between tools instead of between clouds. You wouldn’t expect carpenters to list their experience with a hammer (even if it’s useful for all kinds of projects): instead you want to know what they can build.

Importing existing resources

I can’t even begin to tell you how to import resources with AWS CloudFormation, and neither can AWS’s engineers.

This is pure hyperbole. Importing resources into CloudFormation is documented, and the console has a basic wizard.

If we ignore the exaggeration, there is indeed still room for improvement. While all new resource types should support importing, there is still a long tail of old resources that are not yet supported. “All resources should support resource import” is a valid point. However, CloudFormation was usable before the introduction of this feature. Whether this is a dealbreaker will depend on how frequently you expect to import unsupported resources.

Service Quotas

The list of quotas for the AWS CloudFormation service is just hilarious.

I don’t think that documented limitations are necessarily a bad thing, and I wonder how many of those quotas people actually run into. It’s been a long time since I had to look at that page.

TL;DR

CloudFormation makes different trade-offs than Terraform. Looking at its sharp edges without considering the advantages they bring is unfair.

Use the tool that matches your requirements. If you want safe deployments, tight integration within AWS, and minimal operational overhead, CloudFormation will probably fit the bill.

The post #CloudartotheRescue: DO use AWS CloudFormation (a response) appeared first on Cloudar.

How we do this

We can determine an AWS account by taking advantage of the new S3:ResourceAccount Policy Condition Key. This condition restricts access based on the S3 bucket an account is in (other account-based policies restrict based on the account the requesting principal is in).

If we start with access to an object and write a condition with an “Allow” on exactly one account ID, we can determine if this bucket is in this account. We will be able to access the bucket if we get the account ID right, but we will see an access denied if we try the wrong number.

Reducing the search space

Finding exactly the right account might seem impractical, with literally one trillion possibilities to try, but we don’t need to test them all.

Conditions in policies use string matching and support wildcards. We can leverage that to exclude a whole range at a time. For example, to find the first digit, we can test “1*”, “2*”, “3*” etc. Once we gain access, we know our first digit and can do the same for the second, reducing the number of requests (in the worst case) from a trillion (10^12) to one hundred and twenty (10*12).

Making the requests

Now that we know which policies to test, we need a way to try them.

If the object is public, we have many options since any principal can test these conditions. 

If we have to use a specific principal to access the object, we could change that user or role’s policy. Editing policies isn’t ideal as these are eventually consistent, and we would have to wait after each change.

If the principal is a role, we can take advantage of an assume-role call’s extra parameters to get strongly consistent changes:

The default way to create temporary credentials from a role is a command like aws sts assume-role --role-arn arn:aws:iam::123456789012:role/name --role-session-name ben. That will give us an access key, secret access key and session token with that role’s full permissions. If we add the --policy parameter to that command, we also get temporary credentials, but the permissions will be the intersection of the role’s permission and the supplied policy. This gives us a way to have an additional condition without editing anything in our account. Doing that between 10 and 120 times will eventually reveal the account ID, and that is what the tool does.

Is this a problem?

It’s never a good idea to make discoverable any information that doesn’t need to be. That’s why we replaced all the account IDs in this blogpost with 123456789012. 

However, we don’t consider the account ID to be secret information either. There can be many reasons to share an account ID with an external party, and there are more ways to find it out anyway. Your protections should assume they’re public anyway.

Can I see when someone does this?

The STS actions all happen in the account of the person trying to find the account ID, which will not be visible to the bucket owner’s account. 

S3 data events (the kind of API calls used to test the policy) are not logged in CloudTrail by default. This can be enabled, but it comes with an extra cost, making it probably unrealistic to do so for buckets that get many requests. Alternatively, you can enable server access logging on each bucket, which is much more economical.

When enabled, you should be able to see multiple denied requests in a short time. However: it’s trivial to do this over a longer period, and you will only see that the request was denied (not that there was an abnormal condition).

And if you still see it, there is still a chance that it was someone from your organization who didn’t remember which account a specific bucket was in, which has happened before! With luck, this can help people with that problem!

If you’d like to know more, get in touch with Cloudar today!

The post Finding the Account ID of any public S3 bucket appeared first on Cloudar.

Installation

To use this new feature, you need the latest version of the CloudFormation cli. You can install this with pip install cloudformation-cli (tip: you can use pipx to keep your python environment clean). If you already had an older version installed, run pip install --upgrade cloudformation-cli

Starting a new project

To create a new module, we follow a similar process as with Resource Providers. For this post, we’re going to create a module to set up an S3 Bucket and a CloudFront Distribution for static website hosting.

We first create a new directory, after that we run cfn init. This will ask us for input, and we can specify that we want to create a module for our generic static website

~$ cfn init
Initializing new project
Do you want to develop a new resource(r) or a module(m)?.
>> m
What is the name of your module type?
(<Organization>::<Service>::<Name>::MODULE)
>> Cloudar::Generic::StaticWebsite::MODULE
Directory /cloudar_generic_staticwebsite/fragments Created
Initialized a new project in /cloudar_generic_staticwebsite

Writing our module

We already had a generic CloudFormation template that was doing a similar thing. So we deleted sample.json and replaced it with our own website.yaml. Officially, only json is supported, but in our test yaml worked fine. Since this is a test, we’re okay with taking the unsupported path. For use in a production environment, we can use cfn-flip to convert our template to json.

Depending on your template, you might not have to change anything. Going over the resources to make sure they have clear names and running cfn-lint isn’t a bad idea though. You can see the template we created on GitHub.

Deploying our module

We already had our AWS credentials configured for the AWS cli. Which meant that the only thing we had to do for our deployment was running cfn submit --region us-east-1. If you are submitting multiple times, you can add --set-default to change the default version after submitting it.

You can see the deployed version, with all its properties in the CloudFormation console. This is also a great place to see all the resources our module will create

Using our module

This is where it gets really interesting, we are going to use the module we created. We made a template with one resource (see Github for the full template):

Resources:
  StaticWebsite:
    Type: "Cloudar::Generic::StaticWebsite::MODULE"
    Properties:
      DomainName: !Ref DomainName
      HostedZoneId: !Ref HostedZoneId

CloudFormation will expand that for us, and put every resource that we defined in our module in our processed template.

After waiting for our deployment to finish, we can see all created resources and both our original and the processed template in the CloudFormation console.

Next steps

We’re very excited about this new release and are looking forward to seeing where this might lead. We also have a list of things we think might improve this even more (like an AWS::ModuleName pseudo-parameter), and will probably start conversations about that on the roadmap or issue tracker soon.

The post AWS CloudFormation Modules: A first look appeared first on Cloudar.

Interactive Video Service

The first (and newest) streaming option in AWS is Interactive Video Service (or IVS), which we can summarise as “your own Twitch”². If you use this, you will be using some streaming software (like OBS) on a machine locally to convert your camera source (or screen share,  movie, …) and push it to AWS. IVS is interesting to consider as a first option, as the AWS Elemental suite has a lot more options and assumes a bit of a video-production/broadcasting background.

One caveat with software-based solutions is that you need to get the video on a computer. For a webcam, this is easy. For a stand-alone camera, you will need a “capture card”. These come in different styles with different prices. The most common inputs are HDMI and SDI³ (used mainly in professional equipment). You might get this over USB or as a PCIe card. Prices range from ~20 USD (sometimes good quality, sometimes terrible) to ~200 USD (and more if you need more inputs or better quality).

Elemental MediaLive

The other significant AWS option is the AWS Elemental suite. We will assume that Elemental MediaLive will be our destination. Like IVS, it’s made for streaming video. MediaLive is more flexible in the sources that it supports. It can get content from software that pushes to AWS or from an endpoint where it will request the data (pull). Note that we’re still talking about software, not camera’s at this point. The list of supported formats is here.

If your camera can output one of those options (take a note of whether this should happen in a VPC), you could use MediaLive directly. OBS has an RTMP output option, and MediaLive can accept an RTMP stream as input, so that could be one software solution to try out. You still have the capture card problem at this point (the chance of a non-ip-camera⁴ outputting something usable is slim).

Elemental Link

You’ll notice two “special” formats/inputs on the list of supported inputs: MediaConnect and Elemental Link. We’ll skip MediaConnect in this explanation. You use that if you have an “encoder” locally.

That brings us finally to AWS Elemental Link. Link is a dedicated hardware device to be the equivalent of the capture card + software (+ place to run the software) in one convenient, silent, package. You connect it to power, network, and HDMI or SDI, and it is preconfigured to stream to MediaLive in your account. It also gives you some control over the device from within the AWS console.

Conclusion

As we have seen in this blog post, you do not need Elemental Link to get video into AWS. There are different options for different situations. Sometimes a software solution might be better suited. Other times. the remote management and fanless operation might make AWS Elemental Link a better fit.

The post Streaming video to AWS. Do you need Elemental Link? appeared first on Cloudar.

This blog post will highlight something you might not know yet in four different domains: writing templates, securing deployments, usability, and extensibility. Hopefully, this allows you to do even more with CloudFormation

CloudFormation Linting

One of the best ways to increase your CloudFormation productivity is to minimize the time you’re waiting on a stack that failed to deploy. A great way to do that is to use cfn-lint to make sure you don’t start the deploy if there are errors in your template.

Other tools like cfn-nag, cfn-guard, or taskcat will help you write secure, compliant, and region-independent templates, respectively. Cfn-lint is the one that will save you the most time when authoring templates by flagging misconfigurations, invalid properties, and misformatted templates. It also is easy to integrate into your IDE, pre-commit, or a CI/CD pipeline.

If you’re already using it, I don’t need to convince you of its value. I do, however, have some tips for getting the most out of it.

• You can include experimental rules with --include-experimental.
• You can add your own checks (written in python) with --append-rules
• You can enable more built-in rules with --include-checks I. The I stands for info. One of these informational checks that I find very useful is a warning if a resource that might contain state has no UpdateReplacePolicy/DeletionPolicy.

Another cool feature is that the linter can create a resource graph from your template. This is used by the tool to indicate when you write a circular dependency. It’s just as useful to resolve those circles or create a visual representation of your template.

IAM Integration

There are also some features of CloudFormation itself that I want to highlight. The first is how you can leverage its integration with IAM to prevent people from skipping the infrastructure-as-code step without completely locking down the account.

One way to accomplish that is to use a CloudFormation Service Role tied to your stack. This way, you can give minimal permission to your users and more broad permissions to CloudFormation. If you’re using ServiceCatalog to deploy infrastructure, you’re probably doing something very similar. This solution comes with some IAM complexity. The person creating the stack needs the iam:PassRole permission, but once set up, everyone who can update the stack will do so with that Role assumed by CloudFormation (whether you want that depends on your use case).

If you do not want to think about the effects of iam:PassRole, you can use the aws:CalledVia IAM condition in the IAM policy of the person using CloudFormation. Being an IAM condition comes with its own set of challenges, but you might be more familiar with those.

You use this method by specifying that certain actions are only allowed if the aws:CalledVia key is present and set to CloudFormation, and IAM will deny every request going directly to the service. The documentation has a helpful diagram
.

Parameter Validation

When using CloudFormation to deploy infrastructure, you can make it easier to put in the right parameters by using AllowedPattern and AllowedValues. Both of these only consider one parameter at a time. There is a way to validate Parameters that have a relationship with other parameters, though. Because you might want to validate that two parameters are mutually exclusive or that if one parameter is set, another parameter also has to have a value.

This is probably the most hidden CloudFormation feature because it’s documented in the Service Catalog documentation. And even though the Rules section seems to be a Service Catalog feature, you can use it directly in CloudFormation too. I wrote more about that in a previous blog post.

Resource Providers

And this blogpost would not be complete without mentioning resource providers. Giving a full example of writing one would be its own post. Still, I’d like to show how this successor to custom resources has a lot of potential.

Resource Providers work similar to Custom Resources, and you will not see a big difference in your templates. They should be easier to manage, though (once they can be deployed with CloudFormation – the CloudFormation team is actively working on that). When you write your own Resource Provider, you also implement and expose a Read and List handler. This new approach gives you almost the same capabilities as a native CloudFormation resource.

By creating resource providers, modern features like drift detection and resource import work with your own resources – and will work with every new resource that AWS releases support for.

There is also an integration with AWS Config for your Resource Providers, and AWS is developing more and more resource providers in the open.

Go build

We looked at 4 different ways to improve your CloudFormation experience, and I’m looking forward to hearing your favorite tips and tricks. Good luck making things!

The post Level up your CloudFormation usage with these 4 tips appeared first on Cloudar.

Unfortunately, the blog post did not include a list of changes. Luckily there are other places where you can find those changes (if you know where to look):

For the quick summary of the changes, you can head over to the AWS Architecture Blog: Announcing the New Version of the Well-Architected Framework.

To find more details, with some additional context about why they changed, you can check out a different blog post for each pillar:

• What’s New in the Well-Architected Operational Excellence Pillar? on the AWS Architecture Blog
• Updates to the security pillar of the AWS Well-Architected Framework on the AWS Security Blog
• What’s New in the Well-Architected Reliability Pillar? on the AWS Architecture Blog
• What’s New in the Well-Architected Performance Efficiency Pillar? on the AWS Architecture Blog
• What’s New in the AWS Well-Architected Cost Optimization Pillar on the AWS Cost Management Blog

If you want to see the actual changes, you would have to download both the old and the new PDF and put the questions side by side. And that’s exactly what we did. There is value in seeing the actual changes – even though the posts above already give a decent idea of what is different – but keep the following in mind when you look at the file we compiled:

• We only looked at the titles of the best practices. These aren’t the only improvements, there have been a lot of changes in the descriptions, examples, and explanations too.
• Since the attached document only shows the changes, it’s recommended to use it side-by-side with the official pdf.
• It’s not the prettiest looking document, but we hope it’s useful anyway.

That being said, we’re interested to hear what you think of these changes!

Download the list of changes here

The post The detailed updates to the Well-Architected Framework appeared first on Cloudar.