Mount AWS S3 on Kubernetes pods without Access Key & Secret Key

AWS EKS is a powerful managed Kubernetes service that simplifies the deployment and management of containerized applications. It’s a popular choice for running container workloads in the cloud. AWS S3 is a highly scalable, cost-effective, durable, and secure object storage service.

As we know, containerized applications may require access to external storage for data persistence, combining the power of EKS and S3 can be a game-changer, as it allows you to seamlessly access and store data. This would be really helpful for GenAI use cases, where you need to access the large LLM models which you would like to have inside your container. Instead of copying these large models into the docker images, we can have it on s3 and mount the s3 during the container startup which would make the data available as a network share. In this blog post, we’ll explore how we can mount AWS S3 buckets on pods running on AWS EKS, without using any access key and secret key.

First of all, we need to make sure that the EKS cluster has an existing IAM OIDC provider. In order to check that, first we need to extract the cluster OIDC provider ID, we can use the below command for the same.

aws eks describe-cluster --name cluster_name --query "cluster.identity.oidc.issuer" --output text

Once you have the cluster OIDC ID extracted from the above command, run the below command to check if the IAM OIDC provider is available for the cluster.

aws iam list-open-id-connect-providers | grep EXAMPLED539D4633E53DE1B716D3041E

If we get an output from the above command, then you already have an IAM OIDC provider for your cluster. If no output is returned, then you must create an IAM OIDC provider for your cluster. We can use the below command to create the IAM OIDC provider.

eksctl utils associate-iam-oidc-provider --cluster cluster_name --approve

Once we have validated the IAM OIDC provider for the EKS cluster, we need to create an s3 bucket. I already have an s3 bucket created for this demo. Now we need to create an IAM policy that will have access to your s3 bucket. Below is a sample policy that I used.
NOTE: Just for demo purposes, I’m using the “s3:*” and resource * permissions, however, for actual production use cases, you should limit the permissions to resource-specific.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "VisualEditor0",
			"Effect": "Allow",
			"Action": [
				"s3:*"
			],
			"Resource": "*"
		}
	]
}

Let’s save the above policy as “iam-policy.json”, now we can use the below command to create the IAM policy –

aws iam create-policy --policy-name "aws-s3-mountpoint-policy" --policy-document file://iam-policy.json

Once the policy has been created, we need to create an IAM role and attach this policy. After that, we need to create a Kubernetes service account and annotate the IAM role arn to the service account. The easiest way to do all these is via eksctl command line. For this demo, we will be using a Kubernetes namespace “s3-mount”, we can use the below command to create the k8s namespace

kubectl create ns s3-mount

Once the namespace has been created, we can use the below command to create the k8s service account “s3-mount-sa“, IAM role and then annotate the IAM role arn to the service account.

eksctl create iamserviceaccount --name s3-mount-sa --namespace s3-mount --cluster <cluster_name> --role-name <s3 role name> --attach-policy-arn <arn of aws-s3-mountpoint-policy> --approve

As we can see in the below screenshot, the service account “s3-mount-sa” has been created in “s3-mount” namespace, and the IAM role “aws-s3-mountpoint-role” has been annotated

We can validate the IAM role and policy from aws console as well

NOTE: We could have attached the s3 access policy to the EKS node group role as by default same permission gets propagated to all the pods running inside that node group. However, as a best practice, we should restrict the permission to a specific pod, therefore we are using IRSA(IAM role for service account) here.

Now, we can create a pod that will use the above-mentioned service account, we can use the below manifest to create a pod that will run in a loop and won’t exit

apiVersion: v1
kind: Pod
metadata:
  name: s3-mount-pod-02
  namespace: s3-mount
spec:
  serviceAccountName: s3-mount-sa
  hostNetwork: true
  containers:
    - name: app
      image: amazonlinux
      command: ["/bin/sh"]
      args: ["-c", "while true; do echo $(date -u) >> /example/out.txt; sleep 5; done"]
      securityContext:
        privileged: true

Let’s save the above manifest as “s3-mount-pod.yaml” and use the below command to create the pod.

 kubectl apply -f .\s3-mount-pod.yaml

Now, we can see that the pod has been created and is in a running state.

Let’s exec into pod in order to mount the s3 bucket, we can use the below command to do the same.

kubectl exec -it -n s3-mount s3-mount-pod-02 -- sh -c "clear; (bash || ash || sh)"

Once we are inside the pod, let’s try to create a directory, where we will mount the s3 bucket. For this demo, I’m creating a directory named “s3-mountpoint”. We can run the below command for the same –

mkdir -p "s3-mountpoint"

We can validate if the folder has been created by the “ls -lah” command.

Now, we need to install a couple of tools along with the s3-mount before we can mount the s3 bucket on this pod. We installed unzip, wget, util-linux-ng, awscliv2, and mount-s3 tools.

yum install unzip -y
yum install wget -y
yum install util-linux-ng -y
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
./aws/install
wget https://s3.amazonaws.com/mountpoint-s3-release/latest/x86_64/mount-s3.rpm
yum install -y ./mount-s3.rpm

Once the installation is completed successfully, we are all set to mount the s3 bucket. For this demo, we have a s3 bucket “aws-s3-mountpoint-eks-01” which consists of two files. We can use the below command to mount the s3 bucket inside the pod

 mount-s3 aws-s3-mountpoint-eks-01 /s3-mountpoint

Once the bucket is mounted, we can validate if the same s3 files are available.

These are the same files that we can see on the AWS console of s3.

Note: If you are using custom CNI with AWS EKS, the instance metadata service may not be reachable and you will observe a timeout issue. I have written a blog post on how to fix that, check here

Now, let’s try to do some unzip operation and see if that works on this s3 mount. For this demo, I have uploaded a zip file logs.zip to s3.
I used the unzip command to unzip the log file and it worked as expected. You can see below it’s unzipped the log files

Now, If I go to the AWS S3 console, you can see the same log files are available.

This confirms that after mounting the s3 bucket, we can do some file operations as well. This will be very useful and handy for many scenarios in k8s.

Now, for an actual use case, we can create a dockerfile that will install all the required tools and mount the s3 bucket at the container startup. We can control the bucker name etc. with the env variable in order to have a generic docker image.

I hope this has been informative for you. Please let me know in case of any queries.