Tag Archives: devOps

DevOps Days Five Years Later…Where’s the Security?

It’s been five years since the first DevOps Days in Ghent. Last week, Patrick Debois, the founder of DevOps Days brought the conference back to Belgium for a five year reunion. Let me start off by saying, I didn’t attend the event. In fact, I’ve never attended a DevOps Days event at all. My hometown of Washington, DC hasn’t hosted an event yet. Though a few of my friends and colleagues in the DC/Baltimore area have been talking about organizing one, I haven’t participated. Right now…it’s all talk. So feel free to give me grief if you like.

I’ve been fascinated with the DevOps movement since its inception. I was one of the lucky few who caught wind of the Velocity Conference from the beginning and was able to be an annual attendee. I was a longtime performance engineer and huge fan of the work of Steve Souders and the Yahoo Exceptional Performance team. When I heard there was a conference about Web Performance and Operations, I jumped at the possibility of hanging out with colleagues who spoke the same language as me and shared similar thoughts. Up until then if you wanted to talk software performance, you either went to JavaOne, Oracle World or CMG. Finally, here was a conference all about web performance and operations…and I was immersed in it.

During my time at Velocity over the years, I met a ton of folks who were more than Front-End performance engineers. They were full-stack engineers interested in performance. They were interested in monitoring and measurement. They wanted to do more automation, provisioning and deployment. They were interested in meeting others like themselves who were dealing with the same kind of issues. They were at the conference for the Culture and Sharing if anything.

Velocity is what birthed DevOps days and we owe this to its original leaders (Steve Souders, Jesse Robbins, Tim O’Reilly and many countless others). We equally owe the rise of the movement to Patrick Debois for creating a blueprint for the community to come together in a peaceful and collaborative way via DevOps Days.

Now that I’ve set the stage, let me tell you what I’m actually thinking in terms of DevOps and Security…

Let’s Look At the Numbers

The data I’m going to present below is not highly scientific. It started as an anecdotal thought and moved into me parsing through 5 years of DevOps Days agendas and presentations. I started with the following hypothesis:

DevOps is becoming a real thing. It’s not a fad. It’s not just for startups and unicorn companies out of Silicon Valley. The Enterprise world has accepted the movement and is trying to be a part of it.

I’ve been part of the enterprise software community for years during my time at Blackboard and prior with a Supply Chain software company called Manugistics. What I know about the Enterprise community is that they care a lot about security (application, information and infrastructure security) as well as compliance. I then added the following to my hypothesis:

If the enterprise world accepted the DevOps movement, then most likely it was because the DevOps movement was thinking beyond Automation and Monitoring. It wasn’t just a Culture and Sharing love fest. There had to be more thoughts around securing the enterprise and scaling the enterprise.

So I decided to dig into some data to see if this hypothesis could be proved out. Could I make a correlation from the data? I decided to use the data from every program linked on the DevOps Days website. The only exception was Nairobi, which didn’t list their program. For that, I will conclude that they didn’t have a good program as the S in CAMS is “Sharing”. Since they didn’t share, they get a 0.

The chart below is pretty straightforward. The blue bar represents the number of presentations that referenced Security in some capacity. I looked at titles, abstracts, blogs and even the presentations. I’m sure someone can dig deeper than me and correct the data if they see a glitch. I would certainly welcome the research and would make corrections. The red bar represents the number of sessions. Note this accounts for any Ignite sessions called out on the program. It doesn’t account for hacksessions or BOAF fun stuff.

The last column on the far right is the DevOps Enterprise Summit this past October in San Francisco. I had just attended it and was excited that security was starting to be talked about. I enjoyed the conference immensely, but happened to walkaway thinking we have a long way to go before security becomes a whole track in the DevOps movement.


Here are my initial takeaways. 2012 was the best year in terms of count. I’m a big believer it’s because (3) of the sessions were in Austin where a ton of AppSec and InfoSec players are making waves such as James Wickett and Nick Galbreath. They just happen to be (2) of the (3) presenters on Security. They also happen to be the main folks on the circuit talking about Security and DevOps, other than my co-worker at Contrast Security and one of the co-founders of the Rugged DevOps movement, Jeff Williams.

The number of DevOps Days has jumped dramatically in 2013 (all-time high) and is still close to 3x more than any other year during 2014. I would bet that 2015 will probably have 10+ if not more.

I assumed that as the number of DevOps Days increased, so too would the topic of Security. It seemed like an obvious conclusion in my mind. It’s just not happening. Forget that the first 3 years didn’t really have anything Security related. It was probably called-out, but nothing serious. The numbers show that the percentage of presentations (even the raw numbers) just aren’t increasing.

I wonder if similar conclusions could be made about Security in other conference settings and meet-ups associated with DevOps themes, as well as tooling to support the DevOps movement and Continuous Delivery. Is the problem with the Security community being slow to participate in the DevOps movement, or is it the DevOps community doesn’t know what to do with the Security community?

I honestly thought there would be more penetration of security within DevOps, as well as more penetration of DevOps within security. I had heard references to DevOpsSec. DevOps is real, but DevOpsSec isn’t there yet. The two are flirting and possibly going on a couple of dates, but it’s safe to say they aren’t going steady yet.

I talked about this with my colleague Jeff Williams. He eloquently described the situation between about Security and DevOps in the following sentence. “Security is still an island.” Jeff is right. People are talking about security in the context of DevOps and CD, but not in an emersed and integrated way.

I guess that’s what 2015 will all be about.

Why Am I Even Thinking About This

Let me start off by saying that this was not intended to be a controversial blog. Looking over now, it hopefully doesn’t read too controversial. I had just come from Gene Kim’s conference on Enterprise DevOps and I was starting to think more fluidly about DevOps, the Enterprise and Security. I was glad to see that some folks are talking about DevOps and Security.

There’s still not enough conversation. We need some more mind sharing in the space. We don’t necessarily need it all to come from Security experts. We need folks in the DevOps and CD spaces to jump into Security in a similar manner they did years ago with Performance and Scalability. It makes me think about my own passage to DevOps.

A few years back I was fortunate to watch one of my earliest mentors, Bernie Wong make the move from being a Performance Engineering practitioner to a Security Engineer. There are a lot of similarities between the two practice areas. Bernie convinced me that if I worked hard enough and embraced the Security community and all its good habits and flaws, then it would be an easy transition. I was fortunate to team up with a fantastic Security practitioner at Blackboard, Stephanie Tan to evolve an entire practice area around application security. I learned the ways of application security.

Stephanie and I worked on the problem of application security as part of an enterprise software product, as well as our SAAS cloud products for four years. We talked daily about how to make application security something “continuous” as part of our commit/build/test pipeline in which our tools could give our developers feedback within minutes of a commit. We also thought/worked the deploy and operations piece of live software, but not to the same degree as the commit pipeline.

We looked at every tool on the market. We built some tools as well and open sourced them. We committed to other projects via Pull Requests. We struggled to get a comprehensive tool set and workflow in place at a reasonable cost and without considerable security training. It’s safe to say that Security was an island. We had to run a parallel commit pipeline just of the AppSec team that couldn’t fail the build. We had a couple touch points with the developer commit pipeline, but not to the degree we wanted and visioned.

Alas, we both parted from Blackboard this past summer (amicably and on our own accord) to pursue deeper interests in the security community with other companies.

In full disclosure, I got the opportunity to work on the problem of Continuous Application Security for a company called Contrast Security. I’m not a co-founder, but I’m an early arrival. We evaluated Contrast about a year before and were really impressed. When they created the opportunity for me to join, I couldn’t refuse. It’s an opportunity to tackle Security and CD.

I welcome comments/feedback or tweets to anyone that wants to help make Continuous Application Security and the topic of Security within the DevOps community relevant. Also, the commentary in this blog are solely of mine and do not necessarily reflect the views of my current employer, Contrast Security or my past employer, Blackboard Inc. 

Test Driven Ops at Brewster

I wasn’t able to go to the DevOps in DC Meetup for November. Apparently there was a good session delivered by a guy named Ric Lister from Brewseter.com about how they use Ruby to test their applications. Their main application appears to be an IOs app that acts like a contact aggregation tool. It looks like he talked mainly about two tools: Front-End Loader and Auger.

About Front-End Loader

Front End Loader is a Ruby DSL for declaring load tests. It works in the spirit of tools like JMeter, by simulating a number of users performing a scripted set of actions and displaying metrics about response times and error rates as the requests are performed. Unlike GUI tools like JMeter, however, front_end_loader makes it very simple to declare your requests and to pass data between requests, by looking at the responses to gather data.

About Auger

The Auger library implements a ruby DSL for describing tests to be run against remote applications on multiple servers. The gem includes ‘aug’, a multi-process command-line client.

The primary goal of Auger is test-driven operations: unit testing for application admins. The library can also be used as a framework for implmenting automated tests.

These are the sorts of questions auger can answer:

  • is port :80 on my application webservers open? does /index.html contain a response tag that we know should be served from a given backend data source?
  • is redis running? is it configured as a master? a slave?
  • is elasticsearch responding on all my hosts it should be? what’s the cluster state? do I have the number of data nodes responding that we’re supposed to have?

Clearly a lot of this information includes things you should be graphing. What auger wants to do is give you a quick overview of current status: green == good, red == ruh roh!