Application Load Balancer - Drop Invalid Headers

24 April 2025

We use HTTP headers all the time. In requests we do, we often send cookies, authorization headers with bearer tokens, content length, we get responses with new cookies, expiration dates for caching. We don't think too much about this, even when building things like a REST application. However, it is a nice point for malicious actors to try exploiting our vulnerabilities. That's why today I want to demonstrate you how "drop invalid headers" feature was implemented in AWS Elastic Load Balancer.

Code for this post available on GitHub

The mysterious setting

When creating an ALB in AWS, not many of us think about this setting. We mostly focus on health checks, SSL certificates, security groups, draining timeout. And this setting just sits there untouched. According to Aqua Sec, lack of this setting is makred as high risk AVD-AWS-0052.

However, it is very easy to enable. What we need to keep in mind is that our application might not like it if we didn't conform to standards. To enable this feature in Terraform we need to just add this third line:

resource "aws_alb" "ALB" {
  name                       = "my-load-balancer"
  security_groups            = [data.aws_security_group.alb_sg.id]
  drop_invalid_header_fields = true
  ...
}

Let's build a simple ALB that will just return a fixed response. In order to do this I prepare a Terraform that I will deploy with OpenTofu. As this is just a test, there's no need for SSL certificates or HTTPS listener - the secure one will work the same way as plaintext listener.

In order to create an Elastic Load Balancer, we need a VPC in our AWS environment (including public subnets, Internet Gateway, route tables, etc.) and a security group that will allow us to connect to the ELB. For simplicity, I will use here aws-ia/vpc module that will allow us to construct almost all the components with minimal work required (and make it cleaner than any LLM output).

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" { region = "eu-west-2" }

module "vpc" {
  source  = "aws-ia/vpc/aws"
  version = ">= 4.2.0"

  name       = "alb-example"
  cidr_block = var.vpc_cidr
  az_count   = 2

  subnets = {
    public = { netmask = 24 }
  }
}

Now in a new file we can define our load balancer with a security group, listener and a fixed response. Useful for us will also be the output with the URL we can visit to see the response.

resource "aws_security_group" "alb_security_group" {
  name   = "alb-security-group"
  vpc_id = module.vpc.vpc_attributes.id

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_alb" "sample" {
  name               = "sample-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb_security_group.id]
  subnets            = values(module.vpc.public_subnet_attributes_by_az)[*].id
}

resource "aws_alb_listener" "listener" {
  load_balancer_arn = aws_alb.sample.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type = "fixed-response"
    fixed_response {
      content_type = "text/html"
      message_body = "<html><body><h3>Fixed response!</h3></body></html>"
      status_code  = "200"
    }
  }
}

output "alb_dns_name" {
  value = aws_alb.sample.dns_name
}

For now the load balancer should just reply to us with some HTML. We can test it in the browser as well as using curl in our Terminal. The latter will be later useful to provide some invalid headers. Let's see if the ALB is at least functional.

# Or simply copy and paste the URL after http://
$ curl http://$(tofu output -raw alb_dns_name)
<html><body><h3>Fixed response!</h3></body></html>

Good to see that we have a response. We can turn on verbose mode to see everything that is happening under the hood, all the headers going around and HTTP status code.

$ curl -v http://$(tofu output -raw alb_dns_name)
* Host sample-alb-1234567.eu-west-2.elb.amazonaws.com:80 was resolved.
* IPv6: (none)
* IPv4: 3.11.255.255
*   Trying 3.11.255.255:80...
* Connected to sample-alb-1234567.eu-west-2.elb.amazonaws.com (3.11.255.255) port 80
> GET / HTTP/1.1
> Host: sample-alb-1234567.eu-west-2.elb.amazonaws.com
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< Date: Sat, 19 Apr 2025 07:13:47 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 50
< Connection: keep-alive
< 
* Connection #0 to host sample-alb-1234567.eu-west-2.elb.amazonaws.com left intact
<html><body><h3>Fixed response!</h3></body></html>

Trying invalid headers

I will try now sending some headers. One of them should be invalid. I will not turn on the setting so that all the headers should be accepted.

$ curl -v \
 -H 'X-Custom-Header: valid-header' \
 -H 'X-!NotValid@Header😭: very&badlyformatted:query' \
 http://$(tofu output -raw alb_dns_name)
* ...
> GET / HTTP/1.1
> Host: sample-alb-1234567.eu-west-2.elb.amazonaws.com
> User-Agent: curl/8.7.1
> Accept: */*
> X-Custom-Header: valid-header
> X-!NotValid@Header😭: very&badlyformatted:query
> 
* Request completely sent off
< HTTP/1.1 200 OK
< Date: Sat, 19 Apr 2025 08:13:46 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 50
< Connection: keep-alive
< 
* Connection #0 to host sample-alb-1234567.eu-west-2.elb.amazonaws.com left intact
<html><body><h3>Fixed response!</h3></body></html>

Everything went through, no errors reported. Let's see what will happen if we turn on "Drop invalid headers". Should we see an error or is it just a filter?

resource "aws_alb" "sample" {
  name               = "sample-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb_security_group.id]
  subnets            = values(module.vpc.public_subnet_attributes_by_az)[*].id
  drop_invalid_header_fields = true
}

$ tofu apply
...

$ curl -v \
 -H 'X-Custom-Header: valid-header' \
 -H 'X-!NotValid@Header😭: very&badlyformatted:query' \
 http://$(tofu output -raw alb_dns_name)
...
* Connected to sample-alb-1234567.eu-west-2.elb.amazonaws.com (3.11.255.255) port 80
> GET / HTTP/1.1
> Host: sample-alb-1234567.eu-west-2.elb.amazonaws.com
> User-Agent: curl/8.7.1
> Accept: */*
> X-Custom-Header: valid-header
> X-!NotValid@Header😭: very&badlyformatted:query
> 
* Request completely sent off
< HTTP/1.1 200 OK
< Date: Sat, 19 Apr 2025 08:17:30 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 50
< Connection: keep-alive
< 
* Connection #0 to host sample-alb-1234567.eu-west-2.elb.amazonaws.com left intact
<html><body><h3>Fixed response!</h3></body></html>

Exploring access logs

Still, we don't get any error. Let's do another thing - I suggest enabling ELB logs to see if they actually say anything about dropped headers. For this I need an S3 bucket where I will store the logs and a policy that will enable ELB to put new objects. I will then direct our load balancer to do this. You should look up your region here: Attach Bucket Policy

locals {
  elb_account_id = "652711504416" # ELB account of eu-west-2
}

data "aws_caller_identity" "current" {}

resource "aws_s3_bucket" "logs_bucket" {
  bucket = "logs-bucket-5535325"
}

# This policy should restrict the path to which ELB can write because if we just
# put '*' in Resource, it will allow anyone to use our bucket for their ALB
# logs.
resource "aws_s3_bucket_policy" "log_delivery_policy" {
  bucket = aws_s3_bucket.logs_bucket.bucket
  policy = <<-EOF
  {
    "Version": "2012-10-17",
    "Statement": [
        {
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::${local.elb_account_id}:root",
            "Service": "logdelivery.elasticloadbalancing.amazonaws.com"
        },
        "Action": "s3:PutObject",
        "Resource": "arn:aws:s3:::${aws_s3_bucket.logs_bucket.bucket}/AWSLogs/${data.aws_caller_identity.current.account_id}/*"
      }
    ]
  }
  EOF
}

resource "aws_alb" "sample" {
  name               = "sample-alb"
  ...
  access_logs {
    bucket  = aws_s3_bucket.logs_bucket.bucket
    enabled = true
  }
}

# Adapt bucket name, account number, region and date
$ mkdir -p logs
$ aws s3 cp --recursive \
 s3://logs-bucket-5535325/AWSLogs/999900000011/elasticloadbalancing/eu-west-2/2025/04/19/ \
 ./logs/
$ cd logs
$ for f in *.gz; do gunzip $f; done

I have collected some samples with both requests with proper headers and invalid ones. You can't see the actual headers in the logs but based on the size difference we can find which ones contained additional headers - the larger ones with 193 bytes input definitely had extra headers whereas the smaller ones of 111 bytes in "received_bytes" field did only contain default ones. (For reference the fields in the logs are the following: Access Logs Syntax).

http 2025-04-19T08:42:56.351174Z app/sample-alb/c1aabbcc22334455 89.255.255.255:50804 - -1 -1 -1 200 - 193 190 "GET http://sample-alb-12345678.eu-west-2.elb.amazonaws.com:80/ HTTP/1.1" "curl/8.7.1" - - - "Root=1-68036210-464146a9138538586ff5e14e" "-" "-" 0 2025-04-19T08:42:56.351000Z "fixed-response" "-" "-" "-" "-" "-" "-" TID_9b351616fe757a4eb05b32904a9d92e2

http 2025-04-19T08:43:01.041946Z app/sample-alb/c1aabbcc22334455 89.255.255.255:50806 - -1 -1 -1 200 - 111 190 "GET http://sample-alb-12345678.eu-west-2.elb.amazonaws.com:80/ HTTP/1.1" "curl/8.7.1" - - - "Root=1-68036215-0636826c46a6680554d0583d" "-" "-" 0 2025-04-19T08:43:01.041000Z "fixed-response" "-" "-" "-" "-" "-" "-" TID_29c56d96bfdd254f9feda0b2495f58d5

http 2025-04-19T08:43:02.840550Z app/sample-alb/c1aabbcc22334455 89.255.255.255:50807 - -1 -1 -1 200 - 193 190 "GET http://sample-alb-12345678.eu-west-2.elb.amazonaws.com:80/ HTTP/1.1" "curl/8.7.1" - - - "Root=1-68036216-478dc44c3c867e922e33854f" "-" "-" 0 2025-04-19T08:43:02.840000Z "fixed-response" "-" "-" "-" "-" "-" "-" TID_5bec2e45efdb2d47a359a47534333ef1

http 2025-04-19T08:42:59.706809Z app/sample-alb/c1aabbcc22334455 89.255.255.255:50805 - -1 -1 -1 200 - 111 190 "GET http://sample-alb-12345678.eu-west-2.elb.amazonaws.com:80/ HTTP/1.1" "curl/8.7.1" - - - "Root=1-68036213-54037c2e02539c33533406e4" "-" "-" 0 2025-04-19T08:42:59.706000Z "fixed-response" "-" "-" "-" "-" "-" "-" TID_eadfeace8e8bd444a93cf651d68004e7

As you see there's no indication of bad headers whatsoever in the access logs. But important to note is that these feature is not designed to secure Elastic Load Balancer itself, rather to make sure that no exploits in this field are passed to the upstream services.

Viewing the headers sent

Because of that, I will construct a Lambda function that will just be the default target of our load balancer. It will simply respond with a page and print out all the headers directly. Let's build it with Python.

def lambda_handler(event, context):

    headers = event.get('headers', {})
    headers = [
        f"* {header_name}: {header_value}"
        for header_name, header_value in headers.items()
    ]
    body = "\n".join(headers)

    return {
        'statusCode': 200,
        'statusDescription': '200 OK',
        'isBase64Encoded': False,
        'headers': {
            'Content-Type': 'text/plain',
        },
        'body': body
    }

I will also use Terraform with archive provider to pack and upload the Lambda. There are also some IAM permissions that are required. First, each Lambda needs an IAM role. I will grant "AWSLambdaBasicExecutionRole" allowing to send logs to CloudWatch which will help us debug if necessary. What is more we need to let Elastic Loadbalancing to execute the function.

resource "aws_iam_role" "lambda_role" {
  name = "lambda_alb_role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [ {
        Action    = "sts:AssumeRole"
        Effect    = "Allow"
        Principal = { Service = "lambda.amazonaws.com" }
      } ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_basic_execution" {
  role       = aws_iam_role.lambda_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

data "archive_file" "lambda_zip" {
  type                    = "zip"
  source_content          = file("${path.module}/lambda.py")
  source_content_filename = "lambda.py"
  output_path             = "${path.module}/lambda.zip"
}

resource "aws_lambda_function" "headers_lambda" {
  filename         = data.archive_file.lambda_zip.output_path
  function_name    = "display-request-headers"
  role             = aws_iam_role.lambda_role.arn
  handler          = "lambda.lambda_handler"
  runtime          = "python3.12"
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256

  timeout     = 15
  memory_size = 128
}

# Permission for ALB to invoke Lambda
resource "aws_lambda_permission" "alb_invoke" {
  statement_id  = "AllowALBInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.headers_lambda.function_name
  principal     = "elasticloadbalancing.amazonaws.com"
}

# Target group with Lambda
resource "aws_lb_target_group" "lambda_tg" {
  name        = "lambda-headers-tg"
  target_type = "lambda"
}

resource "aws_lb_target_group_attachment" "lambda_tg_attachment" {
  target_group_arn = aws_lb_target_group.lambda_tg.arn
  target_id        = aws_lambda_function.headers_lambda.arn
  depends_on       = [aws_lambda_permission.alb_invoke]
}

In order to test that all the headers are passed, I will need to first disable the drop_invalid_header_fields option we activated previously in the ALB. Only then our experiment will make any sense. Also I will swtich the default behavior of our listener to forward all the requests to Lambda target group instead of a fixed response.

resource "aws_alb" "sample" {
  name                       = "sample-alb"
  drop_invalid_header_fields = false
  ...
}

resource "aws_alb_listener" "listener" {
  load_balancer_arn = aws_alb.sample.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type = "forward"
    target_group_arn = aws_lb_target_group.lambda_tg.arn
  }
}
...

Viewing the headers

After we have deployed, we can perform some cURL requests to the load balancer. This is just a plain GET request with no additional headers. It just prints all the headers as we expect it to.

$ curl http://$(tofu output -raw alb_dns_name)
* accept: */*
* host: sample-alb-1234567.eu-west-2.elb.amazonaws.com
* user-agent: curl/8.7.1
* x-amzn-trace-id: Root=1-6809c847-59e13c41052857933ee95d5d
* x-forwarded-for: 5.5.1.1
* x-forwarded-port: 80
* x-forwarded-proto: http

So now what we can do is pass a valid header and then try passing invalid ones. I will perform some cURL requests again with the headers and we will see what is the result.

$ curl -H "Test-Header: valid" http://$(tofu output -raw alb_dns_name)
* accept: */*
* host: sample-alb-1234567.eu-west-2.elb.amazonaws.com
* test-header: valid
* user-agent: curl/8.7.1
...

$ curl \
  -H "Test-Header: valid" \
  -H "Test@\$\!Header:: : \!\$Invalid*" \
  -H "Emoji👌Header: 😂should-be-bad" \
  http://$(tofu output -raw alb_dns_name)
* accept: */*
* emoji👌header: 😂should-be-bad
* host: sample-alb-1234567.eu-west-2.elb.amazonaws.com
* test-header: valid
* test@$!header: : : !$Invalid*
* user-agent: curl/8.7.1
...

As we can see, all the headers are passed unmodified - all the characters are received by Lambda. In most cases this is not a problem but everything depends on how we implement the application logic. In order to reduce the risk area, let's again drop the invalid ones.

resource "aws_alb" "sample" {
  name                       = "sample-alb"
  drop_invalid_header_fields = true
  ...
}

This time the invalid headers are nowhere to be seen. Only the clean ones are passed forward to the Lambda function. The drop_invalid_header_fields option applies only to the header names. In the second example below, you can see that passing any characters after the colon is supported. Header names can contain only alphanumeric characters and dashes (not even underscores or dots!). If we add more colons, only the first one counts, others are treated as part of the value.