At Tangocode, we have been relying on AWS ECS and Fargate for some time to run tasks that we can not run on AWS Lambdas. After mastering our configuration and CI/CD, We focused on securing our containers. With this blog, I am going to share some considerations to keep your containers on ECS with the right security level.
Do Not Hardcode or Bake the Secrets
Deploying secrets into containers was the first difficulty we found. Of course, the first solution to make things work was to put the secrets into the Dockerfile and build the images. However, exposing the secrets this way is not ideal, and it opens a huge vulnerability.
To avoid exposing the secrets, we put them in an S3 bucket, and any time the containers run, the first thing they do is to download them. To keep secrets secure in its location, you can do this:
- Configure the bucket to keep all the objects encrypted.
- Restrict the access to the bucket to make sure only the containers with a certain role will be able to execute the permitted actions on the bucket.
S3 Policy Example: With this policy, all the files will be encrypted server-side, an encryption setting has to be used to upload any new file, and the encryption keys will be managed by S3. Also, the first policy statement ensures that only the entities holding the role will be able to execute a Get operation on the bucket:
Rely on AWS Identity Access Management (IAM)
Use the Least Privilege security principle and limit the access that your containers have to your AWS resources by creating an IAM role with limited permissions, so your containers only access the AWS resources that are supposed to. Imagine if an attacker takes control of a container that has ADMIN access to your AWS resources.
Make this configuration by specifying the taskRoleArn on the task definition. Another IAM role required is the executionRoleArn. This role is used before your container is running and allows ECS to make calls on your behalf to deploy the container. For the executionRole Amazon ECS provides the policy AmazonECSTaskExecutionRolePolicy that supports the use cases such as pulling a container image from ECR, use awslogs or awsfirelens drivers.
With this configuration, the containers don’t need to have the AWS secrets to access any resources through the AWS SDKs.
Task Definition Example:
Secure the Container’s Networking
Option 1 – Hide your containers from the Internet
Containers usually need access to the Internet to use third-party services. For example, Payment gateways, databases, etc.. Also, unless you configure VPC end-points, your containers need the Internet to use AWS resources such as Dynamo tables, S3, etc..
The challenge here is to block any incoming traffic from the Internet and allow the containers to make requests to these services that require Internet access. To accomplish this set-up, these are the steps to follow:
- Change the assignPublicIp from the ECS service configuration to DISABLED.
- Create a private subnet in AWS and Route Table.
- Create a NAT gateway
- Configure the Route Table with the NAT.
- Configure the ECS containers to use the created subnet. (ECS requires two subnets as a minimum)
Based on the previous diagram. If an elastic load balancer is needed, this can be inside or outside of a private subnet. If inside, it won’t be visible to the Internet and might require another service on top of it, such as an API Gateway.
Option 2 – Block the incoming traffic with Security Groups
A quicker option to avoid creating a private subnet is to use security groups and limit the incoming traffic. If you are using a load balancer you can configure the ECS service security group to accept incoming traffic only from the load balancer. To do this you only need the security group id used by the load balancer to configure the inbound rules of the ECS service
ECS Service Security Group Inbound Rules:
On the other hand, if the container still needs internet access, but doesn’t need to respond to any request coming from the Internet. You only need to block any incoming traffic in the ECS Service security group.