What Is AWS Lambda Event Filter and How Does It Work?

In November of 2021, AWS announced Lambda Event Filtering – a new way to filter event content at source. Learn all about Lambda Event Filtering in this article.

During this year’s re:Invent, AWS announced a new feature meant to simplify Lambda event processing: Event Filtering.

This means that developers can have their Lambda Function process a subset of the incoming event stream without having the source owner to make any changes.

Using this new feature, developers define Filter Rules which get compared with incoming events. Filter Rules can be as simple as checking that a certain field in the incoming object contains a certain value, or contain complex and/or expressions with numerical range value comparisons (i.e. greater than, less than).

In other words, your Lambda function will only get invoked when the corresponding event passes through the filter successfully.

What’s even neater is that Lambda will automatically disregard or drop any incoming events that don’t match the pattern. This means you can save cost in the long run by avoiding pointless invocations.

Currently, the feature supports the following event sources: SQS, DynamoDB and Kinesis.

To understand a bit more about how powerful this feature is, we need to first take a look at how developers used to achieve this same functionality in the past.

Before Event Filtering – Option 1 – Dropping Messages

Prior to event filtering, developers had to rely on a pretty rudimentary approach. The best we could do at the time is add an if statement as one of the first lines of our lambda function to determine if the corresponding event is one we’re interested in. Something along the lines of…

if (event.key != 'VALUE_I_AM_INTERESTED_IN') {
    return;
} else {
    process(event);
}

In an architecture using SNS to publish the messages via a topic, and a consumer using a SQS queue and a Lambda function, it would look a little something like this:

The big problem with this approach is the unnecessary cost of executing your lambda functions needlessly. In the example above, incoming events can be one of two orderTypes – PURCHASE or REFUND. If we’re only interested in PURCHASE events, we’ll need to instrument our Lambda function to check for the PURCHASE value very early on in the function’s execution before proceeding, and returning if the value is REFUND.

You may enjoy this article on AWS SQS vs SNS vs Eventbridge – When to Use What?

Recall that the primary factor in the Lambda function cost formula is the number of invocations. Using this method, we’re pretty much throwing our hard earned money down the toilet by wasting it on invocations that we’re not actually interested in.

The good news is that a couple years later, an improvement was made to SNS topics to alleviate (but not completely solve) this problem.

Before Event Filtering – Option 2 – SNS Event Filter

A couple of years ago, the SNS team released a feature called Event Filtering. The feature allows subscribers to only receive a subset of messages from the topic based on the content in the message itself.

The criteria for defining the subset could be text matching, numeric value matching, and other methods. The filter rule would then be run against the content of the message. Only messages that met the defined criteria would flow to the subscriber.

For example, if I was an SNS topic owner that published two types of messages, PURCHASE or REFUND, I could specify a filter that will only send messages containing PURCHASE orderTypes to certain subscribers.

In this model, the initial subscription to the SNS topic from a client is tied to the filter, and therefore any message that matches the rules defined in the filter will be sent in the subscriber.

Using SNS Event Filters, our architecture would look a bit something like this:

One of the big problems with this method was that the SNS topic owner would need to make a code change to add support for this feature.

In a large organization, making code changes to support client features can often get de-prioritized or put on the back burner.

In an ideal world, it should be the client that has the power to define which types of messages it would like to receive.

Herein lies the advantage of Lambda Event Filters…

Now With Lambda Event Filters

Lambda Event Filters change this relationship around and put the power into the hands of the clients.

Using event filters, clients now have the ability to define the characteristics of the messages they are interesting in. They can define a set of criteria defining the message rules using what is called a Filter Rule.

Filter rules are similar to SNS Filter rules in the sense you can use complex string matching, boolean logic, numeric value matching, and so on. A full table of supported Filter Rules can be seen in the table below.

Tying this back to our original example where we were only interested in receiving PURCHASE messages, we would event a new Filter Rule on our event source that looks like this:

{
    "orderType": ["PURCHASE"]
}

Simple, isn’t it? This is a pretty trivial example but you could see the potential in this type of mechanic.

Our final architecture using Lambda Event Filters looks like the following:

Lambda Event Filters are currently available for use. You can learn more about them by taking a look at the AWS documentation.