Amazon EKS introduced Pod Identity in 2023 as a feature that streamlines the process of configuring IAM permissions for Kubernetes applications. Pod Identity simplifies the identity management of applications running on top of Amazon EKS by allowing you to set up permissions directly through EKS interfaces, reducing the number of steps required and alleviating the need to switch between the EKS and IAM services. Pod Identity enables the use of a single IAM role across multiple clusters without updating trust policies, and it supports role session tags for more granular access control. This approach not only simplifies policy management by allowing the reuse of permission policies across roles, but also enhances security by enabling access to AWS resources based on matching tags.
Background
When your applications that are running on top of Amazon EKS require sensitive information like credentials to access a database, or a key to authenticate through an API, we call this kind of information secrets. You can secure, store, and manage secrets in AWS Secrets Manager. ASCP allows Kubernetes applications to securely retrieve the secrets stored in Secrets Manager and Systems Manager Parameter Store. Previously, ASCP relied on IAM roles for service accounts (IRSA) for authentication. Although IRSA provided improvements over previous methods, Pod Identity offers even greater security and simplicity.
Pod Identity provides a more secure and efficient way to integrate IAM roles with applications running on EKS, granting more granular AWS permissions to individual Pods, so that you don’t need instance-level credentials or IRSA.
This integration provides the following key benefits over using IRSA:
Enhanced security: Pod Identity provides a more granular and secure way to manage permissions at the Pod level, alleviating the need to expose the IAM role annotation on Kubernetes ServiceAccount objects.
Simplified configuration:Pod Identity streamlines and simplifies the setup process, reducing the potential for misconfiguration, especially in high-scale environments.
Improved operation: Pod Identity reduces the operational overhead compared to previous methods, centralizing the management with the AWS API.
Native EKS integration: Pod Identity is the new standard for IAM integration for applications running on EKS and provides a more cohesive experience.
Solution overview
With this integration, ASCP uses Pod Identity to authenticate and authorize access to AWS services. When a Pod requires access to a secret, the workflow is as shown in Figure 1.
Figure 1: The workflow performed by the Pod Identity and ASCP integration to provide access to a secret stored on AWS Secrets Manager for a Pod running on Amazon EKS
The workflow is as follows:
A user creates an IAM role and a Pod Identity association between the IAM role and the Kubernetes ServiceAccount assigned to the Pod.
EKS API validates the Pod Identity association, allowing ASCP to use this role to authenticate with AWS services.
If authorized, the Pod Identity agent allows ASCP to assume the IAM role assigned to the Pod through the use of a ServiceAccount token.
The Pod retrieves the requested secrets values and makes them available to the Pod through the use of a mounted volume.
Prerequisites
You need to have the following prerequisites in place in order to implement this solution:
Implement the solution
This guide presents two scenarios: single-account setup and cross-account setup. Complete the single-account steps in Account A before proceeding to the cross-account configuration, which involves Account B. The cross-account setup builds on the single-account foundation to demonstrate secure secrets management across AWS accounts.
Amazon EKS cluster setup
Before you start, you’ll need to set up an Amazon EKS cluster with the required add-ons in a single account (Account A).
(Optional) Use the following commands to set environment variables and create an Amazon EKS cluster:
kubectl --namespace=kube-system get pods -l "app=secrets-store-csi-driver"
You should see this expected output:
NAME READY STATUS RESTARTS AGE
csi-secrets-store-secrets-store-csi-driver-fcxf6 3/3 Running 0 41s
csi-secrets-store-secrets-store-csi-driver-j9wqh 3/3 Running 0 41s
Now that you’ve set up your Amazon EKS cluster, in this use case, you’ll create an AWS Secrets Manager secret in the same account where the cluster and the applications reside (Account A).
Create a secret in AWS Secrets Manager with tags kubernetes-namespace and eks-cluster-name within the same AWS account as the EKS cluster:
In the preceding example IAM role permission policy, the use of conditions with kubernetes-namespace and eks-cluster-name tags helps enforce fine-grained access control by specifying that secrets can only be accessed by Pods from specific namespace and clusters. This allows secretsmanager actions only if the secret is tagged with the matching kubernetes-namespace and eks-cluster-name values.
Notice that the preceding example IAM role trust policy allows the Amazon EKS Pod Identity service (pods.eks.amazonaws.com) to assume the role and tag the session. These actions are necessary for Pod Identity to function correctly, enabling Pods to securely access AWS resources.
Finally, apply the role and trust policy:
aws iam create-role --role-name ascp-podidentity --assume-role-policy-document file://trust.json
aws iam put-role-policy --role-name ascp-podidentity --policy-name ascp-podidentity --policy-document file://policy.json
Note the ARN of the new role. You will use it in the next step.
Create a Kubernetes ServiceAccount and add the Pod Identity association between the ServiceAccount and the IAM role:
Create a SecretProviderClass to use the newly created secret in your Amazon EKS cluster.
cat << EOF | kubectl apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: aws-secrets-manager
spec:
provider: aws
parameters:
objects: |
- objectName: "secret-a" # Secret name or ARN to be mounted to the Pod
objectType: "secretsmanager"
usePodIdentity: "true" # Indicator to use Pod Identity instead of IRSA
EOF
Create a new deployment to consume your newly created secret using Pod Identity as a mounted volume.
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: application-a
name: application-a
spec:
replicas: 1
selector:
matchLabels:
app: application-a
template:
metadata:
labels:
app: application-a
spec:
containers:
- image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
name: amazonlinux
args:
- infinity
command:
- sleep
volumeMounts: - name: secrets-store-inline mountPath: "/mnt/secret" # Directory where secret will be mounted to readOnly: true volumes: - name: secrets-store-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "aws-secrets-manager" # Match SecretProviderClass name created serviceAccountName: serviceaccount-a # Specify service account created
EOF
Validate that the deployment was created successfully and confirm the secret was mounted correctly.
kubectl get pods -l app=application-a
kubectl exec -it $(kubectl get pods -l app=application-a -o name) -- cat /mnt/secret/secret-a
You should see this expected output:
NAME READY STATUS RESTARTS AGE
application-a-b98d44bb8-bzvn4 1/1 Running 0 3s
{"user":"user1","password":"passwd1"}%
Working with cross-account secrets through resource policies
For this second use case, you’ll also use the Amazon EKS cluster on Account A, and create a new Secrets Manager secret in a different account (Account B).
In order to access a secret in a different account, you can’t use the default AWS Key Management Service (AWS KMS) key, but will use aws/secretsmanager to encrypt this secret. So you need to first create a new AWS KMS key that allows cross-account access.
On Account B
Create a customer managed key on AWS KMS with cross-account permissions:
Create a SecretProviderClass to use the cross-account secret created in Account B in your Amazon EKS cluster:
cat << EOF | kubectl apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: aws-secrets-manager-cross
spec:
provider: aws
parameters:
objects: |
- objectName: "$SECRET_CROSS_ARN" # Full ARN of the Secret in the ACCOUNT B to be mounted on Pod
objectType: "secretsmanager"
usePodIdentity: "true" # Indicator to use Pod Identity instead of IRSA
EOF
Create a new deployment to consume your newly created secret using Pod Identity as a mounted volume:
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: application-b
name: application-b
spec:
replicas: 1
selector:
matchLabels:
app: application-b
template:
metadata:
labels:
app: application-b
spec:
containers:
- image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
name: amazonlinux
args:
- infinity
command:
- sleep
volumeMounts: - name: secrets-store-cross mountPath: "/mnt/secret" # Directory where secret will be mounted to readOnly: true volumes: - name: secrets-store-cross csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "aws-secrets-manager-cross" # Match SecretProviderClass name created for cross-account access serviceAccountName: serviceaccount-b # Specify service account created for cross-account access
EOF
Validate that the deployment was created successfully and confirm the mounted secret:
kubectl get pods -l app=application-b
kubectl exec -it $(kubectl get pods -l app=application-b -o name) -- cat /mnt/secret/$SECRET_CROSS_ARN
Note that the volume name is the secret ARN, because it’s a cross-account secret.
You should see this expected output:
NAME READY STATUS RESTARTS AGE
application-b-67b755444f-ngrhv 1/1 Running 0 8s
"This is a Cross Account Secret"%
Conclusion
The integration of ASCP with Pod Identity marks a significant step forward in secrets management for Amazon EKS. It offers enhanced security, simplified configuration, and improved operations. We encourage all EKS users to explore this new integration and take advantage of its benefits.
The integration of ASCP with Pod Identity offers these benefits over IRSA:
Simplified setup: With Pod Identity, you don’t need to create and manage service accounts for each application.
Enhanced security: Pod Identity provides more granular control over permissions at the Pod level.
Improved scalability: Pod Identity is easier to implement in large-scale environments.
Consistent AWS experience: Pod Identity aligns more closely with AWS best practices for IAM management.
