Chaos Engineering: A Pretty Organized Affair

Gerhard Jacobs
April 17, 2019

As a tech team, the number of events you end up attending annually can easily morph into a blur of brunches, happy hours and schwag. With that in mind, on the morning of March 28, we hauled ourselves to Workbench, a NYC-based VC fund with a pretty neat – and definitely roomier than expected – venue close to Midtown Manhattan.

From the getgo, however, it became clear that this wouldn't be your run of the mill event. For one, the crowd seemed genuinely engaged and not necessarily as a networking exercise… the room had a studious vibe.

Chaos engineering day 2019 – or #chaosday19 if you’re keen to catch up on the social blow-by-blow – is the fourth of its kind, organized by Casey Rosenthal who's been a member of the chaos community pretty much since its inception.

Casey – who literally co-wrote the book on chaos engineering – cut his teeth as the engineering manager at Netflix, where he managed the traffic and chaos teams; the latter he also helped build. If you’re keen, punch play and have a listen to this interview he did with DevSecOpsdays:


In his own words, “chaos engineering day originally brought people together who helped shape how people run code in production,” but after v4.0 the growing community seems perfectly placed to shape the future of how systems are deployed at scale.


2015

Four years ago the challenge for managers looking to build chaos engineering teams was that, well… chaos engineering didn’t exist – not in its current form anyway. Therein, however, rested an opportunity and so the very fist chaos engineering community day was born; yes, as a means of bringing like-minded people together, but there was a pragmatic purpose as well: to recruit folks who were closely-skilled enough to tackle the tasks at hand.


2019 

Fast-forward to 2019 and we’re a far cry from the initial state. #chaosday19 pulled in 150 people with expertise ranging from security to traffic management. Kicking off at 9am, this year’s lineup included key members of the chaos community who freely shared key insights into their slice of the industry. The super talented Denise Yu (Twitter @deniseyu21) sketched out the highlights of their presentations and we’ve added a few curated social media posts for posterity.



Nora Jones @nora_js

8 Traps of Chaos Engineering


You can't have a prescriptive approach to the unpredictable.

Nathan Aschbacher @gen_nja


Black Tie Chaos: Failing Formally

Nathan spoke about the application of chaos engineering to safety-critical systems like autonomous vehicles. A key take-home was that, as we continually operate systems at greater and varying depths, the growing number of professionals involved in creating a single system ultimately adds to its complexity.

"There’s no value in just running some stuff and seeing it’s on fire. You need to plan."

Charity Majors @mipsy-tipsy

Closing the Loop on Chaos with Observability<


Distributed systems are particularly hostile to being cloned or imitated


Aaron Rinehart @aaronrinehart



Chaos Testing & Security

People operate differently when they expect things to fail.

Kris Nova @krisnova

Reconciling & Chaos

Reconciling is the new regression testing... but better,



Narendra Nalabothula


Chaos Engineering at CapitalOne

Look at failure as a certainty... and an opportunity.

Implementing chaos engineering for financial institutions comes comes with a unique set of challenges and, according to Narendra, resiliency and avoiding single points of failure go hand in hand.



Padma Gopalan - DiRT at Scale

Failure isn't just the loss of computing.



Ashutosh Raina - Madaari for the Monkeys

Failure is inevitable, let's fail in a controlled environment.




Chaos engineering isn’t about one person laboring through complex tests. Nothing grows in a vacuum and chaos engineering is no different, you’re only as good as the team you’re able to build.

As security remains top of mind, a major theme emerging from this year’s event is that security isn’t something to be plastered on afterward, it’s an essential part of the process. As Norah Jones said,

Creating the chaos is easy, thinking about safety is hard.

Finally, as chaos engineering continues to grow in prevalence across major engineering teams, it’s important to understand that no engineering process – chaos or otherwise – happens without planning. Even chaos needs some orchestration and it’s when the world ISN’T on fire that you’re able to experiment.


2020

Seeing the level of discussion both during and in-between presentations it became clear that folks who took the time to attend the 2019 event were genuinely invested in chaos engineering as a discipline; and our team, for one, can’t wait to see what next year’s event brings to the podium.


Request a Demo

Contact Us

Get Pricing