Implementing DNSSEC seems like it should be a no-brainer. After all, why wouldn’t you want to monitor the integrity of DNS traffic? If someone was fiddling with DNS packets en route, wouldn’t you want to stop it?
Yet as any network administrator will tell you, actually implementing DNSSEC on a network isn’t as simple as all that. While the benefits are clear, putting DNSSEC in place isn’t as easy as snapping your fingers or flipping a switch. It’s a meticulous, detail-oriented process that involves sniffing out technical dependencies across the entire enterprise. It also involves cleaning up technical debt that may have been accumulating in the background for years.
Why we implemented DNSSEC
NS1 has long supported DNSSEC for its customers, but until fairly recently, we hadn’t implemented it on our primary service domain. We only had DNSSEC enabled on our marketing and non-critical domains. It was one of those things that everyone knew was important but always seemed to gravitate toward the bottom of a priority list. In the end, the project took a year to complete - not because there was a year’s worth of work, but more because it was a background project that we chipped away at over time.
Beyond the basic need to secure our domains, maintain our innovative edge in the market, and be good internet citizens, there were operational reasons for implementing DNSSEC as well.
In particular, DNSSEC is a make-or-break feature of NS1’s DNS for China offering. Without DNSSEC, NS1 traffic in mainland China would be open to manipulation - a sensitive issue in a market where the government is known to bend DNS for its own purposes. DNSSEC was our way of protecting our customers’ information and ensuring the integrity of our operations.
If we’re being honest, there was some fear involved in this project. DNSSEC is one of those things where you test and test and test, but you don’t really know how things are going to work out in the real world until you turn it on. There’s no such thing as a gradual rollout of DNSSEC. It’s an all-or-nothing proposition, and current industry practices work against you during rollbacks if there are issues. That can be a bit frightening, particularly given the potential blast radius of an NS1 outage.
The road to DNSSEC deployment
We started the project with an overview of our primary service domain. We documented all the dependencies and delegations associated with those zones and records. Beyond simply scoping the exercise, we also wanted to build confidence that the task was doable - that nothing would cause problems with DNS delegation down the line.
Then came the thorny, work-intensive phase of actually sorting through all the records that had made their way into the zone over the course of seven years. Some of those records were easily marked as “keep” or “delete” in five minutes or less. Others took weeks to untangle and adjudicate.
For each record, we started by looking at whether it was in use and active. Then we validated all the sub-delegations (making sure they wouldn’t break once we switched the parent record to DNSSEC signing). This was a particularly arduous part of the project since it isn’t possible to see how subdelegations work with third parties or accounts just by looking at them in the NS1 system.
We did a lot of work in this stage to confirm the spider web of dependencies and connections to map out the chain of trust. We ended up deleting a lot of “dead records” that weren’t connected to anything. We also moved quite a few staging records to different zones - in many cases, cleaning up the zone also meant changes to our operational workflow. This was part of the collateral benefit of DNSSEC deployment - it forced us to be more intentional with the records we put into the zone.
Finally, we tackled the most complex piece of NS1’s infrastructure - alias records and traffic steering configurations. We made quite a few adjustments in this stage to optimize how things were organized, removing shortcuts and transitioning to more textbook deployments that would fit with the requirements of DNSSEC.
In the end, cleaning up zones ended up taking over a month of detailed work. The term “tech debt” is certainly relevant here - that clean-up process felt like we were paying off a significant bill.
The moment of truth
Once the zones were cleaned up, we embarked on an ambitious round of testing and checking. We used DNSViz to examine the chain of trust across our records. We implemented and monitored the use of DNSSEC on our marketing domain, ns1.com, using it as a guinea pig for the larger operational switch.
Even after all those tests, switching DNSSEC on was nerve-wracking. We crossed our fingers and flipped the switch, knowing that we had done everything we could to prepare.
Once DNSSEC was deployed, we kept testing. We used Catchpoint to monitor the user experience in real-time. We continued to use command line tools (dig, drill, etc) and well-known web properties with DNSSEC-enabled queries against our domains to validate that everything came out all right.
In the end, everything went smoothly. All our planning and hard work in the clean-up phase paid off.
Advice for DNSSEC skeptics
DNSSEC is still an underused piece of the DNS protocol. Our internal data shows that just 14% of the queries hitting NS1 infrastructure utilize DNSSEC signing. After a year’s worth of procrastination, detailed clean-up, arduous testing, and sweating out the final deployment, I can certainly understand why network administrators are hesitant to take on a move to DNSSEC. It’s a significant commitment of energy and resources.
Even so, we can all agree that DNSSEC is something that just needs to be done. When and how you do it is something only you can answer. But it’s still a worthwhile effort in the end.
For those who are starting a DNSSEC project, my best advice is to plan, plan, plan. Document everything. Leave no dependency, connection, or routing table unexamined. Make sure the scope of your project is wide enough to capture any detail that could trip you up later (and there are a lot of those, believe me).
It’s also a good idea to enlist outside expertise. NS1’s own customer success team has tons of experience helping customers navigate DNSSEC adoption. With a system that’s this important, it pays to bring in folks who know what to look for and can anticipate possible hiccups down the road.
It’s all about building confidence. With the right scoping, the right plans, and the right testing regimen in place, there’s no reason to believe that you can’t be successful in your DNSSEC deployment. This is doable. I’m living proof.
Learn more about how NS1 uses DNSSEC.
#TechnicalBlog