Earlier this year our team released Flamethrower, a pretty neat new tool designed for teams laboring through complex DNS testing processes, who find their current capabilities somewhat limited.
Some background on the project
Two years ago we started working on a new custom DNS server and needed to test particular features, but found that existing tools like dnsperf -- while great -- didn't exactly have everything we needed.
Why start from scratch?
Straight off the bat, we knew that source port distribution was something we needed in our process, but dnsperf uses a single port; so when you’re trying to test realistic scenarios you realize the traffic patterns are different from real clients.
Second, we wanted something with improved TCP support. TCP in the DNS world used to be something of a second-class citizen, so support hasn’t always been great in DNS servers. We needed a tool from the getgo that would allow us to make sure our servers performed the way we expected.
Finally, we needed something that could generate realistic traffic patterns and could integrate with our CI/CD pipeline, because as you’re working on a specific server and introducing new features, you’d ideally like to know how those changes impact performance.
Written in C++14 (17), Flamethrower uses LDNS as the main DNS library to construct packets. Additionally, we use uvw, an asynchronous I/O package -- and a great wrapper for libuv -- that provides a good C++ interface with a way to handle memory allocation safely.
Flamethrower from a user perspective
Image 1 represents a basic example of what the tool does. On the left is a simple invocation of the utility: executing the flame binary with some parameters. In this case, we're sending the same query to server a.ns.example.test repeatedly. We always ask for name foo.example.test and type SOA.
As you can see on the right, Flamethrower presents us with basic info about what’s going on, then a summary of statistics during the test run and then the result in a human-readable form. The SERVFAILresponses in this example are likely due to a firewall blocking the way.
Flamethrower Architecture
We started with a payload generator that prepares the DNS queries to be sent; the traffic generator handles networking, either by creating UDP sockets or TCP sessions and sends queries. The volume of queries interacts with the rate control -- the traffic generator sends queries in batches so we added the rate control function. Finally, Flamethrower has the ability to collect metrics, which is particularly useful for machine processing.
Individual components
Payload generators
Static - used for simple, single query name and type lookups.
File - comparable to dnsperf, gives you the ability to write a text file with record name and record type per line, allowing you to process the same format.
Randompkt - random binary garbage and nonvalid DNS packets.
Okay so these won’t necessarily crash your DNS server, but it’s a valuable feature because it allows you to test your monitoring abilities. What we realized when we were testing our server was that the tooling we use to collect metrics didn’t see these types of queries at all -- because they’re malformed -- and the parsers in the metrics collectors didn’t anticipate them as they’re just random junk.
Three similar generators generating valid DNS packets, but with random domain names:
Numberqname - names prefixed with a random numeric label:
1234.example.test, 42.example.test
Randomqname - similar to the first, but with random, possibly binary query names. DNS specifications dictate that in a domain name you can only have denominations like letters, numbers, underscores, but internally you can add binary values in query names.
J4kjx\000zz3d\o64sa.example.test., bo4._kf\042.example.test
Randomlabel - Names prefixed with random, non-binary labels. The Mirai cyber attacks from two years ago followed this format, with their random prefixed attacks. This payload generator allows you to monitor and simulate them.
XRY4HFY9dpltTb.example.test., ady.clo.ZUqV4gji.example.test.
Traffic generator - handles networking
Flamethrower currently runs in a single thread only, so all operations on several sockets are driven by the I/O. Expect about 100k queries per second on a single core, depending, of course, on whether it’s on your laptop or a high-end server. You can run multiple instances of flame at the same time if you want to utilize multiple cores (future versions will make this easier).
Configuration options
-c Number of concurrent traffic generators.
The meaning changes slightly depending on whether you pick TCP or UDP later on in the process; but essentially, if you create for instance 100 concurrent traffic generators it means that for UDP, Flamethrower will make 100 UDP sockets, showing you traffic from 100 random ports. For TCP this will open 100 connections and try to keep them alive, so whenever the server closes a connection, it’ll reopen a new one.
The following options give you the ability to decide on the number of queries to send, as well as the time delay between them:
-q Number of queries sent in a batch
-d Time (delay) between batches
For instance: 1000qps = 10 concurrent generators with 100qps per generator x 10 queries every 100 milliseconds.
Protocol selection option:
-P udp Default. One bound UDP socket per traffic operator.
-P tcp One TCP session per traffic generator. The session is re-established when the server closes the connection.
-P tcptls Similar to TCP but using TLS. This is useful for testing DNS-over-TLS.
Rate Control
Image 3 illustrates how you can cut the rate of queries in case the generator is too fast. The graph shows how you can set the rate control for constant rate or define the “qps flow” so you can set the rate to change over the course of the run. In this example, we set qps at 1000 for 60 seconds and then 100 qps for 60 seconds.
We sampled the rate at 30 seconds, which is why the graph ended up looking like a saw wave instead of the square wave of traffic actually sent by flame.
Metrics
If you run flamethrower with -o it’ll give you the results in a JSON format, so for each second, you’ll have a line of JSON that contains everything you see in the visualization Image 4, but per second and unique to your own configuration. We feed this into Elasticsearch, which will give you a pretty good visualization of your data.
Below, we’ve included a handful of test samples so you can see how Flamethrower performs different ‘asks.’
Going forward, our team has some ideas for how we aim to improve the Flamethrower tool, including:
Multi-processor support
The ability to target multiple servers
Source address spoofing
DNS-over-HTTP and DNS-over-QUIC support
Query rate jitter improvement
The ability to extract query rate pattern from a pcap
Overall performance tweaks
Flamethrower lives in the GitHub DNS-OARC organization, alongside dnsperf and other fine DNS testing tools. We built Flamethrower to test individual DNS elements, but ultimately we’d like to make further improvements both for our own use and for the DNS community at large.