Filter Chain 101

Overview

The Filter Chain is the powerful technology NS1 uses to implement dynamic and intelligent DNS decision making algorithms across your DNS records.

When NS1 gets a DNS query, we look up all potential answers to the query — for example, all your web servers. We then pass those answers through a "chain" of filters, each of which performs a simple operation on the list of answers (i.e. removing, sorting, and so on).

Filters make decisions based on metadata attached to each of the answers, like whether an answer is up or down, how loaded it is — anything you can imagine.

NS1 responds with the answer(s) that emerges from the last filter in the chain. From this simple concept you can build incredibly powerful DNS decision making algorithms, tailored entirely to your application. Combining the Filter Chain with Data Feeds for real-time metadata updates, NS1 gives you the most powerful DNS traffic management toolkit around.

How does the Filter Chain work?

The basic purpose of the NS1 Filter Chain is to make dynamic choices about how to respond to a query for a given DNS record.

The Filter Chain can examine a variety of information to arrive at a decision on how to respond to a query -- most importantly:

  • The collection of possible answers
  • Information ("metadata") about each answer, group of answers, or the record as a whole (e.g., up/down status, geographic information, priorities and weights, etc)
  • Information about the requester (e.g, requester's IP address)

At the highest level, when a DNS query arrives at an NS1 node, the full set of possible answers, metadata, and requester details is dumped into the Filter Chain, and what pops out the other end after the Filter Chain executes is a refined set of answers to be delivered to the requester.

The Filter Chain execution process looks a lot like a Unix pipeline, in which each input passes through a sequence of programs that manipulate and act upon the input, each in a simple and precise manner.  In an NS1 Filter Chain, each Filter does one and only one thing, simply and efficiently. A Filter Chain is a sequence of filters acting upon the list of potential DNS answers for a query: the output from one Filter is passed as the input to the next Filter in the chain.

An easy way to quickly understand how the Filter Chain works is to look at an example. Let's examine a common Filter Chain configuration with 3 filters in the following order: Up, Geotarget Regional, and Select First N (with N=1).

This Filter Chain is enough to run a simple Content Delivery Network! Suppose you have the following answers configured in an A record (say, fff.example.web):

  1. 1.1.1.1, with two metadata fields set: "Up" (set to "true"); and "Geographic region" (set to "US-EAST") -- maybe this server is in Digital Ocean's NYC2 datacenter or some other East coast facility.
  2. 2.2.2.2, with "Up" set to "true" and "Geographic region" set to "US-WEST" -- this is a server in some West coast facility.

You can also connect Data Feeds to each answer -- say, from a monitoring service or our NS1 Monitoring -- to tie the "Up/down" metadata to health checks. We won't cover that here so check out this article about failover for some discussion on how to do that.

Now, let's walk through the query resolution process. Suppose a query arrives for fff.example.web from a DNS recursor with IP address 6.6.6.6. Then the query is resolved as follows:

  1. NS1 loads the configuration for fff.example.web, including the above Filter Chain, collection of answers, metadata, etc. It kicks off the Filter Chain with an initial answer list like: [1.1.1.1, 2.2.2.2].
  2. The UP filter executes first. It examines the "Up/down" metadata field for 1.1.1.1, and then for 2.2.2.2. If either is marked "down", the answer is removed from the list. (If all answers are marked "down", the UP filter does nothing -- better to return some answer than none at all!)
  3. The resulting list from the Up filter is passed to the Geogtarget Regional filter. This filter looks up the requester's IP ( 6.6.6.6) in a GeoIP database. Let's say they're in California. The filter then steps through the current list of answers and computes distance between the requester and each answer, using the "Geographic region" metadata field. Since the requester is in California, which is in the US-WEST region, the distance between the requester and 2.2.2.2 is 0. The distance between the requester and 1.1.1.1 is higher since the requester isn't in the US-EAST region. Finally, the filter sorts the answers by proximity to the requester. So, if both answers were "up", then in this case what pops out of Geotarget Regional is a list like: [2.2.2.2, 1.1.1.1] -- a new ordering reflecting proximity to the requester of each answer.
  4. Finally, we want to "pick" an answer to return to the requester. That's what the SELECT_FIRST_N filter does: keep the first N answers from the list, and discard the rest. We've configured our Filter Chain with N=1, so we pick the first answer, 2.2.2.2, and toss out 1.1.1.1.

This process can be replicated for any Filter Chain record by using our new Analyze Filters tool. Here's what that looks like for this example:

This is a really simple example, but it generalizes well. Maybe you add 3.3.3.3 in the "EUROPE" region, and 4.4.4.4 in the "ASIAPAC" region -- the Filter Chain can remain the same: we'll pick the "closest" server to the requester that is up.

If you have multiple servers in each of your datacenters, it's useful to group them together using "regions". You can associate metadata with regions. The Filter Chain resolves metadata by first checking for the value of a field in the answer itself; then the region (if any) to which the answer is assigned; and finally in the metadata for the entire record. All of these metadata "tables" can be managed on the configuration page for the DNS record.

At this point, we've just scratched the surface -- we've shown you a simple 3-filter Filter Chain -- but it's powerful enough to pull off complex logic that with many other DNS providers requires a tangled collection of cascading records, or is simply impossible to do. NS1 has a growing collection of Filter algorithms. You can mix and match any of them together into Filter Chains to pull off traffic management algorithms perfectly tailored to your application.