IAM Roles Anywhere

Jan 14, 2025

I have recently used AWS feature called “IAM Roles Anywhere” in order to access AWS services from another cloud. In this post I’ll outline the process and the experience.

Baseline: IAM Roles for Service Accounts

If you’re running Kubernetes inside AWS, you can use IAM Roles for Service Accounts, where you annotate your Kubernetes service accounts with the desired role, and magic happens.

For example, your K8S service account might look like this

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ray
  namespace: ray
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/ray

And the IAM role might have the following trust policy to allow access from K8S:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111111111111:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/NNN"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.eu-central-1.amazonaws.com/id/NNN:sub": "system:serviceaccount:ray:ray"
        }
      }
    }
  ]  
}

With this definitions, any pod using your service account automatically assume the specified IAM role&mdashvery easy and convenient.

Authorization for external workloads

If your workload run outside of AWS, the most common choice is to create a new user, give it the necessary permissions, and generate access key. However, if that key is compromised, an attacker can use the key with no contraints.

IAM Roles Anywhere aims to improve things. You need to have a public key CA, and you configure AWS to trust that CA. You then issue a X.509 certificate for your workload. There is an API call that allows to request session tokens for a specific IAM role, and if that API call is signed by the client certificate, IAM will accept it and issue the session token that can be used in further API calls.

IAM Roles Anywhere Walktrough

Create a Private CA

To speed up things, I created AWS Private CA. While it costs 400 dollars per months, if you never used it before, the first 30 days are free. I have created a new CA using appropriate names for company and business units, and used ECDSA P384 key algorith and SHA256 ECDSA signature algorithm. When you create a new CA, it has “Pending certificate” state, you must explicitly select “Get CA Certificate” from the actions menu. Finally, record the ARN for the created CA, as we’ll need it below.

Configure IAM Roles Anywhere

In the console, you need to create a “trust anchor” from your private CA, and then a “profile” that is associated with the desired IAM Role. This is well described in the documentation.

Create a client certificate.

Creating a client certificate is just a set of CLI commands

$ openssl ecparam -out ec_key.pem -name secp384r1 -genkey
$ openssl req -out csr.pem -new -key ec_key.pem -nodes -keyout private-key.pem
... input names for certificate ...
$ aws acm-pca issue-certificate \
      --certificate-authority ...CA ARN... \
      --csr fileb://csr.pem \
      --signing-algorithm "SHA256WITHECDSA" \
      --validity Value=365,Type="DAYS"
... outputs certificate ARN ...
aws acm-pca get-certificate \
      --certificate-arn ...Certificate ARN...
      --certificate-authority-arn ...CA ARN... | \
      jq -r '.Certificate' > certificate.pem

Check authentication with the certificate

The AWS SDK does not directly support using IAM Roles Anywhere. You need to run a separate utility, aws_signing_helper that will obtain session token. The utility can be downloaded from the releases page.

Then, run

./aws_signing_helper credential-process \
    --certificate certificate.pem --private-key ec_key.pem \
    --trust-anchor-arn  ...Trust Anchor ARN... \
    --profile-arn ...IAM Anywhere Profile... \
    --role-arn ...Role ARN...

If everything is well, the above should output JSON that can be used for authentication.

Kubernetes setup

Finally, we’re ready to configure Kubernetes workloads.

Create a secret with the certificate:

kubectl -n ray create secret generic iam-ra \
    --from-file=certificate.pem=./certificate.pem --from-file=key.pem=./private-key.pem

Then, deploy the following, admittedly huge, pod definition:

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-config
  namespace: ray
data:
  config: |
    [default]
    region = eu-central-1
    credential_process = /root/shared/aws_signing_helper credential-process
      --certificate /root/.certificates/certificate.pem
      --private-key /root/.certificates/key.pem
      --trust-anchor-arn ....
      --profile-arn ....
      --role-arn ....
---
apiVersion: v1
kind: Pod
metadata:
  name: checker
  namespace: ray
spec:
  containers:
  - name: app-container
    image: public.ecr.aws/aws-cli/aws-cli:2.22.32
    command: ['sh', '-c']
    args: ['aws sts get-caller-identity; echo The app is running! && sleep 3600']
    imagePullPolicy: Always
    volumeMounts:
    - name: certificates
      mountPath: /root/.certificates
      readOnly: true
    - name: aws-config
      mountPath: /root/.aws/config
      subPath: config
      readOnly: true
    - name: shared
      mountPath: /root/shared
      readOnly: true
  initContainers:
   - name: aws-credentials-setup
     image: amazon/aws-cli:2.22.32
     command:
     - /bin/sh
     - -c
     - |
       # Install rolesanywhere-credential-process
       curl -o /root/shared/aws_signing_helper https://rolesanywhere.amazonaws.com/releases/1.0.6/X86_64/Linux/aws_signing_helper
       chmod +x /root/shared/aws_signing_helper
     volumeMounts:
     - name: shared
       mountPath: /root/shared
       readOnly: false
  volumes:
    - name: aws-config
      configMap:
        name: aws-config
    - name: certificates
      secret:
        secretName: iam-ra
        items:
        - key: certificate.pem
          path: certificate.pem
        - key: key.pem
          path: key.pem
    - name: shared
      emptyDir: {}

Here is what we do above:

Create an AWS config, that will use the credentials helper to obtain session
The the pod init container, download the credentials helper
Mount the secret with the certificate, as well as config file, to the main container
Use aws sts get-caller-identity to check for success.

It should be noted that we use a specific version of the credentials helper, as the latest version of the aws-cli docker image and the helper are not compatible.

If everything is OK, the pod will report its newly obtained identity.

Conclusions

While the path above led us to some success, I would say that overall the developer experience here is very convoluted

The number of steps is very high
Instead of using AWS SDK directly, we need an additional binary
There are version coflicts between that library and official CLI images
The changes we need for a pod are large and intrusive

The last point is the most critical—if you have 100 pods, editing them all to add additional volumes or init containers is very complicated, especially as some pods will be created programmatically.

The overall conclusion is that:

IAM Roles Anywhere can be used for small cases of workaloads outside of AWS
It would be painful for large deployments
For large deployments, you probably should investigate OpenID Connect
If you still plan to use IAM Roles Anywhere, you probably need to write your own K8S mutating hook to automatically adjust all new pods.

Hope this helps, and let me know if you have any comments.