AWS IAM Failure from Kubernetes? Missing Roles and IRSA Complexity Explained

Observed Problem

When using Kubernetes to interact with AWS serverless services like DynamoDB, RDS, or Lambda functions, you may encounter IAM-related errors due to missing or misconfigured roles. Common issues include:

AccessDeniedException: User is not authorized to perform: <AWS_SERVICE>:<ACTION> on resource because no identity-based policy allows the action

These errors occur when the appropriate IAM roles are not configured properly, and integrating IAM roles with Kubernetes using IRSA (IAM Roles for Service Accounts) or other identity solutions adds additional complexity.

Analysis

AWS IAM is used to securely authorize access to various AWS services. In a Kubernetes environment, accessing AWS services like DynamoDB, RDS, or Lambda functions requires that the pods making the requests have the necessary permissions. This typically involves using IAM Roles for Service Accounts (IRSA), which allows assigning IAM roles to Kubernetes pods.

Why is This Complex?

• IRSA Complexity: IRSA relies on Kubernetes service accounts linked to IAM roles using a trust relationship. This process involves:

1. Setting Up Trust Relationships: Creating an IAM role and establishing a trust relationship with the AWS OIDC provider, allowing the service account to assume the IAM role.
   
   

2. Service Account Annotation: Annotating the Kubernetes service account with the appropriate IAM role ARN. This annotation allows AWS to authenticate the pod using a projected Kubernetes service account token. This token is distinct from the short-lived credentials returned by AWS STS when the role is assumed with sts:AssumeRoleWithWebIdentity. The AWS SDK within the application dynamically uses the projected token to obtain these temporary credentials, which are used to securely access AWS resources.
   
   

3. Managing Short-Lived Credentials: AWS SDKs within the application use the injected tokens to securely obtain credentials to call AWS services, but this adds extra steps to configure and manage.

• EKS Pod Identity: AWS has introduced EKS Pod Identity (previously referred to internally as IRSA v2) to provide a native way for Kubernetes pods to assume IAM roles. This method is distinct from IRSA and aims to simplify identity management for pods by reducing the manual configuration required. It still involves managing IAM trust policies and annotating Kubernetes service accounts, but it streamlines the process of token handling and authentication compared to traditional IRSA.
  
  

Resolution Workflow

1. Identify the Issue
   
   

   • Check the logs of the pod that is attempting to access the AWS service. You might see errors like:

   AccessDeniedException: User is not authorized to perform: dynamodb:PutItem on resourcebecause no identity-based policy allows the dynamodb:PutItem action

   
   

   • This message indicates that the pod does not have the correct permissions to access the AWS resource.

2. Verify Service Account Configuration
   
   

   • Ensure that the pod is using the correct service account. Use the following command to verify:

   kubectl get pod <pod-name> -o jsonpath='{.spec.serviceAccountName}'

     

   • Confirm that the service account is linked to the correct IAM role.
     

3. Check IRSA Trust Relationship
   
   

   • Verify that the IAM role has a trust relationship set up with the Kubernetes OIDC provider. The trust relationship should look like this:

     

    {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": {
              "Federated": "arn:aws:iam::<account-id>:oidc-provider/<oidc-provider-url>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
              "StringEquals": {
                "<oidc-provider-url>:sub": "system:serviceaccount:<namespace>:<service-account-name>"
              }
            }
          }
        ]
      }

   
   

   • Ensure that the <service-account-name> matches the service account used by your pod.

4. Update IAM Policy
   
   

   • Update the IAM policy to include the necessary permissions for the AWS service. For example, if accessing DynamoDB, your policy should look like this:

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "dynamodb:Query",
              "dynamodb:GetItem",
              "dynamodb:PutItem"
            ],
            "Resource": "arn:aws:dynamodb:<region>:<account-id>:table/<table-name>"
          }
        ]
      }

   
   

   • Attach the updated policy to the IAM role associated with the service account.
     
     

5. Verify the Fix
   
   

   • Restart the pod to ensure it picks up the updated credentials and role configuration:

   kubectl delete pod <pod-name>

   
   

   • After the pod restarts, check the logs again to verify that the AWS service calls are now succeeding.
     
     

How Otterize Can Help

Manually configuring IRSA, IAM roles, and trust relationships can be error-prone and time-consuming. Otterize’s Intents Operator and Network Mapper automate these processes, making it much simpler to integrate Kubernetes workloads with AWS services.

• Automatic Role Assignment: Otterize can automatically detect the required permissions based on the service interactions and assign the appropriate IAM roles.
  
  

• Simplified Annotation: Otterize manages token projection and role assumption seamlessly, ensuring your pods always have the correct credentials without the need for manual configuration.
  
  

• Domain to IP Translation for Network Policies: When working with external AWS services, Otterize can also help by translating domain names into IP addresses for network policies, ensuring that your egress rules are correctly configured.
  
  Here’s a quick tutorial on how to deploy Otterize and start creating Network Policies, and a detailed guide on AWS IAM automation for your EKS clusters.
  
  

Leverage ACK for Seamless AWS Resource Creation

In addition to simplifying IAM integrations, you can also use the AWS Controllers for Kubernetes (ACK) to manage AWS resources directly from your Kubernetes cluster. With ACK, you can create and manage AWS services like DynamoDB, RDS, and Lambda functions using Kubernetes CRDs, further integrating AWS with your Kubernetes workflows. Read more in my blog about ACK here.

Ready to make network policies a breeze?

Stop stressing over network policy details. Let Otterize handle the heavy lifting with ClientIntents, and get back to focusing on your app’s real business.

Stay connected and learn more

If you found this article helpful, we'd love to hear from you! Here are some ways to stay connected and dive deeper into Otterize.

🏠 Join our community

Stay ahead of the curve by joining our growing Slack community. Get the latest Otterize updates and discuss real-world use cases with peers.

🌐 Explore our resources

Take a sneak peek at our datasheets:

• Zero Trust and Microsegmentation

• PCI Compliance

• Cloud IAM automation

Or continue your exploration journey with our blogs and tutorials, or self-paced labs.

Don't be a stranger – your insights and questions are valuable to our community!