Public Kubernetes Services with Web Application Firewall (WAF) on AWS

Purpose

The purpose of this article is to architect and engineer a solution which provides layer 7 security using cloud-native services with public facing services running on Kubernetes 1.7+.

In this post, I hope to create a sustainable solution to solve the problem Arnaldo De Moraes Pereira alludes to in this blog post; which I interpret as, “I want to use AWS WAF to protect my load balancer deployed from Kubernetes.”

Problem Background

Kubernetes (k8s) LoadBalancer services are an abstraction to AWS Elastic Load Balancers (ELBs). Kubernetes does not support AWS Application Load Balancers (ALBs), which are layer 7 load balancers. ELBs are not layer 7 load balancers; therefore, AWS Web Application Firewall (WAF) does not support attaching to an ELB.

AWS CloudFront is Amazon’s content distribution network (CDN) service. AWS WAF can be attached to AWS CloudFront distributions; therefore, at a low cost, we can add CloudFront with WAF. Unfortunately, the ELB can still be accessed directly, which has a solution AWS suggests using. With this solution, CloudFront’s dynamic public IP ranges are constantly added to the ELB’s security group ingress rules.

Unfortunately, me and my team have noticed that at times k8s will recognize the mismatch between the definition’s loadBalancerSourceRanges values and security group ingress rules, and then overwrite ingress rule changes to match the definition (honestly, we are not 100% sure of the trigger, but have experienced this multiple times).

Requirements Considered

This solution would provide an architecture which will meet business needs given the following scenario.

  • The business is running k8s 1.7+ on Amazon Web Services (AWS)
  • You are working on developing a web application or API to be available in the public DMZ
  • For security reasons, it is desired to provide layer 7 web application firewall (WAF) capabilities
  • Users must not be able to bypass the WAF to access the endpoint (in other words, the ELB must only allow ingress traffic from CloudFront)

Note: There are additional steps we can take to secure or optimize this environment, and each area has a fair amount of depth. As examples, we can go deeper into configuring WAF rules, get into alerting on ELB and WAF logs, or look into the benefit/cost of a tool such as AWS Shield Advanced. However, I want to keep this post fairly general.

Visual Representation of Solution

waf.JPG

Note: This solution focuses on SNS, Lambda, Tagged Security Group, CloudFront,  and WAF and its rules. It is recommended to include encrypted S3 for all logs, Shield Advanced, Route 53 for DNS, and routing so workers are not publicly exposed.

Sample CloudFormation Script

The goals of infrastructure as code include DEV/PRD parity, configuration management, version control of infrastructure components, and ease of deployment. CloudFormation is a great tool for maintaining infrastructure as code (IaC). Alternatives to CloudFormation include Terraform (Hashicorp), Ansible (Red Hat), and you can even use Ansible playbooks to deploy your CloudFormation templates! Terraform and Ansible are vendor-agnostic tools which use the same syntax and formatting across various cloud providers, respective to each providers’ resource types. Google Cloud Platform (GCP) and Microsoft Azure both offer their own IaC tools for their own platforms as well.

Ok ok… sorry for the spiel… I think IaC is very important… moving on…

This CloudFormation script (modified from the documentation AWS recommends, above, using the NIST 800-53 framework) creates the following resources (only listing those relevant to this post):

  • CloudFront distribution
  • Web Application Firewall and WAF rules
  • AWS Lambda function which writes ingress rules to particularly tagged Security Groups
  • SNS Topic subscription which invokes the Lambda function whenever AWS updates its public IP ranges (ie. AWS publishes; Lambda subscribes)
  • Security Group particularly tagged (to be used in the k8s service definition)

Sample K8s Service Definition

You must modify a few variables in the sample service definition below. I recommend the annotations, as well as the loadBalancerSourceRanges. The loadBalancerSourceRanges creates a default security group, which we cannot avoid; if we do not specify a value, it is open to 0.0.0.0/0 (no source restrictions; highly discouraged). Therefore, we are locking it down to a single private IP range per RFC1918 in addition to adding the extra security group from the CloudFormation script.

  • {service-name} is the name of your k8s service.
  • aws-load-balancer-ssl-cert is a free ssl cert generated from AWS Certificate Manager which is used for your endpoint
  • aws-load-balancer-extra-security-groups is the security group id that is output from the CloudFormation script in the section above
  • Others, recommended:
    • {logs-bucket-name} is the s3 bucket where the ELB would write its access logs
    • aws-load-balancer-access-log-s3-bucket-prefix is the prefix within the s3 bucket where the ELB would write its logs (assuming it’s a bucket shared for multiple services’ logs)
Example K8s Service Template
apiVersion: v1
kind: Service
metadata:
  name: {service-name}
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: {arn:aws:acm:us-region-1:123456789012:certificate/unique-id-123456790}
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true
    service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: true
    service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: {logs-bucket-name}
    service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: {service-name}/
    service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: {security-group-id}
  labels:
    app: {service-name}
spec:
  ports:
    - name: http
      port: 80
    - name: https
      port: 443
  selector:
    app: {service-name}
  type: LoadBalancer
loadBalancerSourceRanges:
- 10.0.0.0/32
Note 1: I highly recommend using declarative commands to deploy services in Kubernetes such as kubectl apply -f service.yaml; this way, changes do not delete and re-add services, but update them (as opposed to imperative commands such as kubectl {create,edit} -f service.yaml).

Note 2: When deploying this, if you get errors, make sure to check that your worker nodes’ associated Ec2 instance IAM roles have adequate permissions with respect to least privilege.

Cost

Assuming you already have a Kubernetes cluster, and you are using this solution on top of it, the costs would be calculated as follows (according to current AWS pricing data). The variable cost largely depends on your application’s traffic, but for a service which has a low amount of requests, and without caching at CloudFront edge locations, the cost would be less than $30!

  • Elastic Load Balancing
    • $0.025 per Classic Load Balancer-hour (or partial hour)
    • $0.008 per GB of data processed by a Classic Load Balancer
  • Web App Firewall
    • $5 per web ACL per month
    • $1 per rule per web ACL per month
    • $0.60 per 1,000,000 web requests
  •  CloudFront
    • Free for first 50 GB egress
      • Then $0.085/GB/month for first 10 TB egress
    • Free for first 2,000,000 HTTP/HTTPS requests
      • Then $0.01 per 10,000 HTTPS requests
  • Lambda
    • Free for up to 1,000,000 requests (they do not publish to this SNS topic that frequently)
  • Security Group, SNS Subscription
    • Free

Summary

If you find this helpful, awesome! If you have proposed ways to build on this, post your ideas! If you know of an alternative solution, feel free to share it!

Advertisement

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s