So I Did This Podcast…

podcastingI’ve been a long listener to podcasts for a while. My personal favorites of all time have got to be the Ted Radio Hour, Revisionist History, Freakonomics Radio, 30 for 30 Podcast and Real Sci-Fi with Wade and Willy. Of course, I’ve had a few others on my list that over time I just couldn’t take anymore, so I dropped them.

Recently, I was on the receiving end of a special invite to be a guest of Matt Watson of Stackify. I have been a fan of Matt for a couple of years now. I think his company is really impressive and doing more than their share of content marketing to educate the world in the land of APM and Java/.Net communities. Almost every day, I get Chrome notification about a new blog or post on the Stackify website. If you don’t read it already, I strongly recommend you do. When the invite came to be a part of his podcast, it was a no brainer in my mind.

Matt asked me to talk shop about Application Security in his developer podcast series called “Developer Things”. You can hear my appearance if you click on this link. Let me start off by saying, Matt’s a great interviewer. He was able to keep the conversation flowing. He’s perpetually ready to have follow-on questions and he’s just a “pro” about doing research.

It was great to talk shop. Since coming over to Contrast, I haven’t really had the chance to talk about why I made the move and why I’m such a believer in our product. In my last few years at Blackboard, I was super focused on Continuous Integration and quality oriented engineering. I wanted to figure out ways to make telemetry of our applications in development and production into sources of truth when it came to making tough decisions about refactoring and improving code quality.

Screen Shot 2018-02-10 at 2.17.12 PMContrast has been my platform for the last 3+ years to help developers write more secure code and get immediate feedback on commit. When we learn about vulnerabilities and bugs in production, which we more than often do, our Protect product is there to help customers responsibly address their threats in real-time so they can go back and make the “right refactor” and not the “rushed refactor”.

What was cool about doing this podcast, is that for those who know me, my roots are in Performance Engineering, which is Matt’s wheelhouse as well. There are so many parallels between Performance Engineering and Application Security.

The number one parallel in my mind is measurement. I’m specifically referring to measuring code in the context of anti-patterns. As a performance engineer, my number one tool was a profiler. Profilers were great because they were measuring sticks for code. They could tell me how much something cost and could help me visualize an anti-patterns in a simple call stack.

Security profilers, like Contrast Assess essentially do the same thing. The difference is that we don’t focus on frequency and timings. We focus on design principals and anti-patterns in the code. We overlay evidence of security research in the 3rd party libraries that we often use as developers. We can show weaknesses in your code after simply executing it, as well as in the library code you leverage to simplify your development.

There are a ton of other parallels with performance engineering, but probably the 2nd most relevant parallel is that when code can’t scale it’s a bug. When a vulnerability is detected in code, it too is a bug. Performance issues and security threats aren’t black magic. They happen because the logic in one’s code is flawed and exposes a defect. So many developers see performance and security as “not their job”.

So probably the 3rd parallel between performance engineering and application security, is the frustration practitioners face when trying to educate and empower their developer communities. In 2006, AWS launched it’s EC2, cloud compute footprint. I personally didn’t jump on the AWS bandwagon until 2009/2010 timeframe. When I did, I found that my software colleagues using EC2 were very focused on performance. The reason why is so obvious. Every poor design decision or performance anti-pattern, subsequently cost more money.

The same can be said about security. I don’t want every developer to go through a breach in the same vein as a performance issue. There has to be some kind of experience to drive incentives…or so I think.

More to come later. I just wanted to put some thoughts down and share the journey. Keep in touch friends!

-Steve

 

Hiring Engineers: Why We Do Open Book Code Challenges

One of the few topics my friends and colleagues in the industry are willing to talk about are their strategies for recruiting and hiring talented engineers. It’s kind of funny because as an industry of practice, software engineers are so open about everything. Recruiting and hiring is such a competitive exercise, it feels like only the big giants like Google, Facebook and Apple go out of their way to talk about their approaches.

For this blog, I am going to talk about the topic of interviewing candidates. Like most companies, the process takes time and effort. Most candidates often make upwards of two visits to our office. Our average time to go through the process of introduction to offer takes about 4 weeks. In the tech world, that’s like the equivalent of two decades.

It’s Not Just a Job

When I got out of college in the late 90’s, people would jump jobs every 4 to 8 months. If your resume wasn’t littered with startup after startup, or enterprise after enterprise, you weren’t really playing the game right. For a young pup like myself, who watched his father have the same job for 50 years running his own dental practice, it was mind blowing to think the industry approved of this rampant mobility.

boh

I approach things from the perspective that a job isn’t just a job. Rather, a job is an experience in your life. It becomes part of your DNA, just like going to high school or college. Your job consumes 1/3 or more of your time each week, so you better make sure it’s the right place.

We want our candidates to want to be at Contrast. I’m sure everybody says that and wants to believe that phrase to be the gospel. We truly believe it. We want Contrast to be your startup…your top choice. We’ve passed on some really talented engineers that didn’t see much value in our process.

Meeting the Team

I read an article a few weeks back about Etsy which referenced the word “culture fit”. It made me chuckle to say the least. We try our hardest to build a team of skilled engineers who want to work in an environment that values respect for one another and offers the chance to speak open and honestly. We don’t look for clones or brogrammers. We look for engineers that want to make the Internet safer and want to be a part of a high-performing team.

contrast

I think it’s important to sell my team from the start. I want every candidate, whether their an intern or a full-time engineer to trek to our office for a meet and greet. The meeting can take anywhere from 30 to 60 minutes. While it consumes a good amount of my time and pulls my engineers away from their assignments, it’s the first step in establishing a relationship of trust and openness with a candidate.

We are pretty transparent from the start. This initial meeting is not an interview for the candidate, but rather the chance for the candidate to interview us. We want our candidates to come into the office with a boat load of questions to make sure this is the right place to interview.

The Project

Every engineer that’s been hired at Contrast since I started has had to do a project. The project has changed over the years. The link above shows you the current state of the project. The project is open book and candidates do not have a time limit to complete. We often advise candidates to work on it over the course of a week or two at most, otherwise they won’t complete it.

The project is wholly owned by the author. We don’t want our names mentioned in the project. We do ask that the author codes the project and submits it to Github. Some folks are uneasy about a coding project in the public persona. Most like the project as it gives them a reason to build up their public coding profile.

We ask candidates to do projects for a couple of reasons. First, the project is designed to showcase a candidate’s ability to take on new work and see how serious they are about working with us. The really serious candidates crush through the assignment in less than two weeks. Second, the project is used as the focal point for the majority of the interview (see below). Third, the project gives us a chance to see how well you deal with unfamiliar technologies, as well as demonstrate your creativity.

The Interview

Most technical interviews are awkward, frustrating, unrealistic to name a few attributes. Our technical interview isn’t about how great of a developer you are. Rather, it’s about going through the experience of working a project and receiving/giving peer feedback.

We generally perform a small code review of 2 to 4 engineers with a candidate. We once had 12 people in a room with a single candidate, which we really blew. I won’t make that mistake again. Apologies to that candidate as the interview was way too overwhelming.

The code review starts with the candidate walking us through the project. He/she tells us how they interpreted the problem and how they decomposed their work. Next, they walk us through their design and implementation. They are then given the chance to call-out things they would do differently before we ask questions.

The interview team then asks upwards of 4 to 6 questions about the project, design choices, implementation decisions and further elaboration of what’s next. It’s not all questions though…my engineers will always add commentary of what they enjoyed or were concerned about with any project. Most of the time the candidate takes the feedback real well and comes back with a good follow-up question or response. Occasionally, we have a candidate that’s upset with the feedback, but can’t offer a good path forward. When this happens, both parties realize it’s not an ideal match.

The Experience

donnie

The overall experience of meeting the team twice and working on a project can be overwhelming to some who are looking for quick work. For the folks who’ve joined our team, it was a good formula for making the decision about whether this was the right place to work.

It may take time and in our space, time is not unlimited. It’s safe to say that it’s the right time for everyone to make an educated decision about their next job or next hire.

We’re Still At It…

Two years ago, I declared in my last blog entry that I would make “more time” for blogging and low and behold, I failed to do that. Shocker right? It makes sense that my readership dropped to practically none. I think my wife and a few folks for an Amazon Click Farm are my only loyal readership at this point.

I figured that I needed to write an “ice breaker” blog before I start blogging more. I do have more to write and more to say. So I will use this blog as my uncomfortable attempt at pretending I’ve been blogging all along.

The Recap

cattyboh

In my last blog, I teased my massive readership about our move to the historical Natty Boh Tower in Canton. Two years into it, I have to say, every day I love driving to the office and seeing/working with my team. We are a lot bigger than when we moved in back in the summer of 2015. We’ve added 10 more engineers to the office, with plans to add another 4 full-time engineers and 3 summer interns in a matter of weeks. As I mentioned before, the fine folks at Hyperspace outfitted the entire office with Steelcase and Turnstone. I will have to post some pictures in an upcoming blog.

When I started, we only had a handful of customers. I could namely them all and tell you the main point of contacts by their first name. For exactly two years, I wasn’t only the VP of Engineering, but I also served as our primary Tier-1 engineer. Now we have an entire team of support engineers and I’m still learning our newest customers as they on-board.

Initially when I started, we had (1) agent that was GA, our Java agent. We had a second agent that was technically in beta, our .Net agent. We were about to begin our third agent, NodeJS. We had practically no integrations or Open Source contributions.

contrast_sticker

Today, we are putting the finishing touches on our 4th language, Ruby and underway with our next language, which I will keep under wraps until we get closer to beta. We’ve got two modes to our language portfolio: Assess and Protect.

Probably our biggest news since I last wrote, outside of our growth (customers, people and product) would be the investment we received last year from our Series-B. When we took in our Series-A, we were really heads down and quite frankly all working 18-hour days and most weekends. I still work 18-hour days from time to time, but now it’s more a modest 14-hour day 🙂

gc-partners

Our Series-B lead investor is General Catalyst and we are fortunate to have Dr. Steve Herrod on our board. Steve is the former CTO of VMWare and quite frankly the mentor that I needed to challenge and motivate me.

What’s Exciting About the Now

The fun part of my job is that I don’t have to spend hours and hours pushing paper. My last 3 years at Blackboard felt like I worked perpetually in HR and Finance. What I loved about my first 8 years at Blackboard was how hands on I was with our technology. At Contrast, I love that I’m so involved in our development and deployments. It’s tough to juggle being an executive, while at the same time influencing our technology, but it’s worth the late nights and weekends.

We’ve got many parallel projects happening right now, but there are probably a couple of projects that really get my psyched at night to continue working. I’ll give the quick highlights of each project and why I’m enjoying them below.

Serverless (AWS Lambda)

Screen Shot 2017-05-09 at 10.02.42 AM

I’ve been an eager beaver to work with Lamba for over a year now. I didn’t go to reInvent when it was announced a while back, but I certainly did my research after the announcement. As an application guy who is a firm believer in distributed program and agent-based pipeline architectures, the notion of serverless computing seemed mind-blowing.

My colleague, David Hafley, who joined me at Contrast after Blackboard, were both looking for a project or two to get started. We challenged our team to consider leveraging Lambda and forking a GoLang project called Goad to build an elastic benchmark kit for Contrast. Our plan is to release the kit to our customers and speak at a series of summer meet-ups to share the experience. This was only (1) Lambda project. We have a few about to kick-off.

What I love about this project is that we just don’t know how Serverless will land over the next couple of years. If it truly takes off, we have the opportunity to be a part of a crazy ride. The unknown is kind of fun, not only because it’s different, but because it’s unchartered.

Building a Language Agent…Fast…Really Fast

I had the option to hire this one particular engineer either as our Full Stack lead or to go build our Ruby agent with another team member. He and I came to the conclusion that building a Ruby agent would be more fun and beneficial to the company. We’ve built up the agent very rapidly. We’ve had a lot of learning with our first three agents to leverage, as well as a fair amount of shared code (you would be surprised). While we haven’t GA’d the agent yet, we do have the agent running in a few customers’ environment. They are having great success with the agent.

What’s cool is this is really the first agent that I was able to influence how we assembled the team and approached our development. I’m stepping out of my comfort zone in engineering and helping our marketing and sales teams prepare to bring this baby to market.

 

 

Seven Months Into Startup Life…Tell Me Again Why I Haven’t Been Blogging About It?

Yep…dead giveaway with the title. I’m starting my eight month at Contrast Security this week. It’s been pretty hectic, fun, tiring, all-consuming. Is there another word to describe that startup life has basically taken over my life, my wife’s life and even my kids’ lives. I wouldn’t trade it for the world. I figured I would use this blog to catch-up my precious readers of what’s going on.

Hiring Some Great Talent

I waited a few months before I brought in former colleagues shared a lot of my technical and business ideology. I think it was the right thing to do as i established some credibility with my team and formed a good foundation with the team. I did nonetheless bring in some excellent colleagues who worked with me in the past. So far I’ve brought four members from the past teams, as well as two other colleagues from other companies that I valued for their hard work and dedication.

I really should start by saying I really inherited great talent. The folks on my team that were with Contrast when I joined in September were/are very talented. Who says technical talent can exist out of Silicon Valley? I can attest after 15 years in the business that great tech talent can be found all over the world. Keep an open eye and an open perspective…then you will find them.

Move to AWS…It Really Will Make a Difference

The best part about technology is ubiquitous access. Nearly all of our technology that we use and build is cloud-based. In fact, it’s 100% cloud based. I don’t think I would ever want to host technology again given what I know today about costs, management and simplicity.

Right now we are about 30% Atlassian Cloud, 30% AWS cloud, 30% Firehost Cloud and the remaining 10% at various point technology clouds like Office 365, New Relic and Balsami (yes…Balsamiq has a SaaS version of their product). Our goal is to make our AWS investment over 60% of our cloud by summer. Shocking that Amazon still isn’t profitable.

Building an Office

It’s been a long and tireless effort to get us an office space. We started looking at building a space, but settled on a great location in an even cooler building at the Natty Boh Tower in Canton.

nattyBoh

Both co-founders live in Baltimore near me. They have opinions about what they want. I have to say, I really agree with every opinion they have. The opinions kind of vary from our CEO’s opinion btw…He’s out in Palo Alto, the mecca of startups and tech companies. He outfitted a spacious open floor plan in Palo Alto’s industrial district. He was able to get furniture for like 10 cents on the dollar. No such luck in Baltimore. It’s hard to get nice furniture on a dime in Baltimore in bulk.

Don’t get me wrong, you can find items here and there. To find one dozen desks, some chairs and a few conference rooms worth of goodies, is tough unless you make huge compromises like going to Ikea, or somehow stumble across the goose the laid a golden egg.

We were lucky. We found the amazing people over at Hyperspace to help us outfit our furniture. They’ve been great…I gave them a budget and they managed it to a tee.

We haven’t moved into the space yet. We get the keys on Friday, May 1st. We haven’t settled on our furniture either. I’m hoping that happens this week. Once we move in, I’ll be sure to post some pics of the place…

Why It’s Important to Build a Better Product

As an engineer I’ve always felt critical of the various applications I’ve purchased or downloaded for free. If I’m going to take the time to use an application, I really want it to work. I want it to work all of the time. I don’t want to have to be super technical to figure some ridiculous workaround. I certainly don’t want a kludgy experience. I simply want the software to work.

So I have these expectations about the software developers who build the products that I use. I expect them to build the best products. I just assume their code is tested. I assume it scalable and responsive. I maybe take for granted that it’s secure, but with a blind eye I assume it is secure. When I report issues, I guess I expect them to drop everything they are doing and fix my issue. Why do I have those expectations? My attitude is that if your core product doesn’t work correctly, why are you building more product? Shouldn’t your core product work first before you introduce new product?

buggysoftware

Motivation for this Blog

For weeks I’ve been reaching out to various contacts we have at the company and then through my own personal network to find customers who are great at giving us feedback. Feedback is essential for software companies looking to expand and grow. When you are lucky to find someone who will tell you that your product has issues, but they will continue to use it…well those kinds of customers are like the equivalent of gold.

What Features Are Considered Most Important That We Stop Developing New Features?

When I joined Blackboard in 2003, we had a really feature rich product. We had about 30 sub-systems (web apps). Each of those sub-systems had dozens to some cases hundreds of use cases. We had bugs….lots of bugs. Customers reported a lot of those bugs. We had so many bugs that in my first year, we had more support engineers than we had real engineers because we couldn’t keep up with the in-bound calls from customers. Some customers would log 10+ issues a day.

stronbadtechsupport

We knew we had a huge problem on our hands as an engineering team at Blackboard because we wanted to build more features. We didn’t want to really touch old code. We were focused on the new features that would help us get newer, cooler, better paying customers. On the one-hand that’s great to want to get those bigger fish. On the other hand, you have a pool of dedicated customers who quite frankly were your early adopters. They took a risk on us. Some took a small risk financially and some took a bigger risk financially. All have taken the risk of their time in using our product. We need to treat time like money.

I bring all of this up because when we attempted to filter through the minutia of issues, the team came to a collective agreement about two things. We agreed that we would be responsive to customer escalations when issues tied to a set of core features and use cases were sent to support. These core features were our bread and butter features. They were features that absolutely had to be fixed. We had to be responsive, because we knew we simply couldn’t spend the time improving our test coverage. In the early days we followed 3 core principals:

Step 1: Identify our Most Important Features and Capabilities

As a team (Engineering, Product Management and Support) we sat down and categorized every feature and use case in the product. We missed a few here and there, but met again and re-prioritized the list. We defined escalation paths for reported issues that we provided to our Support team to leverage when customer issues were reported.

Step 2: When an Issue Tied to these Most Important Features were Reported…We Agreed to Re-Prioritize Our Work and Fix Them

Since Product Management was in the discussions from the beginning, they understood that new features would sometimes be delayed due to buggy legacy features. They would review the issues coming in as well and would more often then not give the thumbs up to delay a feature in favor of making a core feature more robust and reliable.

Step 3: Always Make the Product Better

As we were forming this 3-step model, every single person chimed in that we should stop development in lieu of test automation. Let me just say this once and only once…”That my friends will never happen…”. Software has to be fluid. We can’t stop producing software and releasing software because it makes our software stale. Customers want small variations of change (not huge changes) on a reliable and consistent cadence. If the train was to stop, getting it back on the tracks is really…really hard.

We need carve out time in three ways:

a) When a customer issue is reported and we fix it…we need to add automated test coverage to minimize re-introducing issues.

b) When a customer reports one issue…we should dig into shared parts of the code and make sure nothing else is broken.

c) When we build new features, we need to incorporate the time to include better automated test coverage.

Bees With Machine Guns

bees

So apparently I’m late to the party. A really cool tool is making the rounds this week on one of my favorite podcasts (Reply All) called Bees with Machine Guns. The kind folks at the Chicago Tribune have created this little app to generate tons of EC2 micro-instances for load-testing. Who needs JMeter, Apache AB, Grinder or even commercial tools like LoadRunner or Soasta when you could get away with a nice little tool like this. The tool is a few years old based on this blog. It’s gone through 2 releases now. It might be a worthy project to make a pull request against.

Take a look at this blog here for a full breakdown of setup and running of the tool.

Time to Start Blogging Again…Hopefully More to Come

I have been meaning to post a blog for quite some time. I’ve been busy growing my new team, building new product and establishing a new identify for me in the AppSec space. It’s been more than rewarding to say the least. So I figured I would put a quick blog for those few readers out there who have wondered where I’ve been all of these months.

Let’s Start with Velocity 2013

I got a call around 6:30am PST on June 21, 2013 from my former boss at Blackboard. She had been my boss for the better part of a decade and she knew I was west coast, so I was surprised to get a call so early. It was one of those quick calls. It lasted 2 minutes. Basically, she said she was leaving Blackboard to move onto another job and that some changes were coming and that I would be meeting my new boss in a couple of days.

Gene Kim and DevCon 2013

A few weeks after Velocity was our own conference in July. I had been planning it for months with a few of my colleagues at Blackboard. I was to co-emcee the Developer Conference with Mike McGarr with special guest Gene Kim as our keynote. The conference was my defining moment at Blackboard. I had given dozens of talks over the years at DevCon and BbWorld, but this was a conference I had been dreaming of putting on for years. It went without a hitch. Mike and I nailed it…our customers were psyched…Gene Kim rocked the house!

Carbon Based Units and Lifestyle Jobs

When we all got back from Vegas, many of us who were part of Bb’s core LMS division realized we were not in Kansas anymore. There was new leadership in town. The company was going to change whether we wanted it to change or not.

I don’t want to go into too many details. From BbWorld 2013 to when I left before BbWorld 2014, I saw some of the best engineers in the world…many who had spent the better half of the decade working with me, decide to pick-up and leave. Most if not all of these engineers worked long days and delivered good work. In the eyes of some, they were compared to CBUs (Carbon Based Units) and their work was considered a lifestyle job because they worked from home or in remote offices.

For those of us who were close to the ground and had an ear on the pulse, we knew that this was a talented bunch of engineers, passionate about their work and loyal to the brand. I saw more engineers walk out the door than walk in. I tried to bring in some new engineers and for a short period, I was successful. In the end, I too succumbed to being miserable and had to pack-up 11 years worth of books and memories.

Summer of Steve

I took the whole month of July off before I would start work again. My wife was kind and patient enough to let me do my own thing. It was the best 3 weeks a guy could ask for. I would start my days taking my kids to swim practice. I would go to the gym or hangout with the other parents. On Monday’s I would volunteer with my daughter’s golf practice. Tuesdays through Thursdays I would volunteer at tennis. Everyday I played golf. I got in 19 rounds over the course of 3 weeks.

The best part is that every day I was excited to wake-up and do it all over again…

6 Weeks of Consulting

I took a 6-week tour for a gig that I will look back upon and say I was a really good consultant. I wasn’t a consultant though…I went to work with one of my longtime colleagues from Blackboard. Sadly, I feel bad that I let him down. The job just wasn’t for me. My expectations were different than theirs. They wanted me to do one thing…I thought I was brought on to do something else…In the end, when Contrast came calling, I realized it was better to cut bait at 6-weeks before it was too late.

Starting Over Has It’s Perks

The “new normal” as my wife called it was all about re-establishing the passion in my career. For 9.5 of my 11 years of Blackboard, I woke-up every day excited to take on the world. Those last 18 months were quite possibly the most painful 18 months I’ve ever gone through. I won’t allow that to happen again…Mark my word 😉

It’s Feels Like I’m 28 All Over Again

The best part of my day starts around 6am when my alarm clock goes off. I love to get up in the morning. I love to look over my left shoulder and see my beautiful wife. I love waking my kids up to get them ready for school. As it relates to me…well, I absolutely love to get going on work. Every new day at Contrast brings a different challenge and an excitement that reminds me of when I was 28 and started to build the PerfEng team at Blackboard.

Growing Our Team…Building Some Cool Products

I was fortunate enough to inherit a good size engineering organization (9 engineers) considering the size of our company. Most of our employees are engineering. We have a rich, engineering culture. I’ve added to it in my short-time. I’ve brought on 4 engineers so far. I have 2 more starting in a couple days to get the count to 6 new engineers. It’s the right size for where we are now. We are lean…nimble…We can do almost anything we want.

A Promise…Of Sorts

It’s been entirely too long since I last blogged. It may be an empty promise, but I am going to do my best to keep blogging. This blog was more for me than anyone…If anyone got something out of it, I’m glad I could help.

DevOps Days Five Years Later…Where’s the Security?

It’s been five years since the first DevOps Days in Ghent. Last week, Patrick Debois, the founder of DevOps Days brought the conference back to Belgium for a five year reunion. Let me start off by saying, I didn’t attend the event. In fact, I’ve never attended a DevOps Days event at all. My hometown of Washington, DC hasn’t hosted an event yet. Though a few of my friends and colleagues in the DC/Baltimore area have been talking about organizing one, I haven’t participated. Right now…it’s all talk. So feel free to give me grief if you like.

I’ve been fascinated with the DevOps movement since its inception. I was one of the lucky few who caught wind of the Velocity Conference from the beginning and was able to be an annual attendee. I was a longtime performance engineer and huge fan of the work of Steve Souders and the Yahoo Exceptional Performance team. When I heard there was a conference about Web Performance and Operations, I jumped at the possibility of hanging out with colleagues who spoke the same language as me and shared similar thoughts. Up until then if you wanted to talk software performance, you either went to JavaOne, Oracle World or CMG. Finally, here was a conference all about web performance and operations…and I was immersed in it.

During my time at Velocity over the years, I met a ton of folks who were more than Front-End performance engineers. They were full-stack engineers interested in performance. They were interested in monitoring and measurement. They wanted to do more automation, provisioning and deployment. They were interested in meeting others like themselves who were dealing with the same kind of issues. They were at the conference for the Culture and Sharing if anything.

Velocity is what birthed DevOps days and we owe this to its original leaders (Steve Souders, Jesse Robbins, Tim O’Reilly and many countless others). We equally owe the rise of the movement to Patrick Debois for creating a blueprint for the community to come together in a peaceful and collaborative way via DevOps Days.

Now that I’ve set the stage, let me tell you what I’m actually thinking in terms of DevOps and Security…

Let’s Look At the Numbers

The data I’m going to present below is not highly scientific. It started as an anecdotal thought and moved into me parsing through 5 years of DevOps Days agendas and presentations. I started with the following hypothesis:

DevOps is becoming a real thing. It’s not a fad. It’s not just for startups and unicorn companies out of Silicon Valley. The Enterprise world has accepted the movement and is trying to be a part of it.

I’ve been part of the enterprise software community for years during my time at Blackboard and prior with a Supply Chain software company called Manugistics. What I know about the Enterprise community is that they care a lot about security (application, information and infrastructure security) as well as compliance. I then added the following to my hypothesis:

If the enterprise world accepted the DevOps movement, then most likely it was because the DevOps movement was thinking beyond Automation and Monitoring. It wasn’t just a Culture and Sharing love fest. There had to be more thoughts around securing the enterprise and scaling the enterprise.

So I decided to dig into some data to see if this hypothesis could be proved out. Could I make a correlation from the data? I decided to use the data from every program linked on the DevOps Days website. The only exception was Nairobi, which didn’t list their program. For that, I will conclude that they didn’t have a good program as the S in CAMS is “Sharing”. Since they didn’t share, they get a 0.

The chart below is pretty straightforward. The blue bar represents the number of presentations that referenced Security in some capacity. I looked at titles, abstracts, blogs and even the presentations. I’m sure someone can dig deeper than me and correct the data if they see a glitch. I would certainly welcome the research and would make corrections. The red bar represents the number of sessions. Note this accounts for any Ignite sessions called out on the program. It doesn’t account for hacksessions or BOAF fun stuff.

The last column on the far right is the DevOps Enterprise Summit this past October in San Francisco. I had just attended it and was excited that security was starting to be talked about. I enjoyed the conference immensely, but happened to walkaway thinking we have a long way to go before security becomes a whole track in the DevOps movement.

devopsDaysSec

Here are my initial takeaways. 2012 was the best year in terms of count. I’m a big believer it’s because (3) of the sessions were in Austin where a ton of AppSec and InfoSec players are making waves such as James Wickett and Nick Galbreath. They just happen to be (2) of the (3) presenters on Security. They also happen to be the main folks on the circuit talking about Security and DevOps, other than my co-worker at Contrast Security and one of the co-founders of the Rugged DevOps movement, Jeff Williams.

The number of DevOps Days has jumped dramatically in 2013 (all-time high) and is still close to 3x more than any other year during 2014. I would bet that 2015 will probably have 10+ if not more.

I assumed that as the number of DevOps Days increased, so too would the topic of Security. It seemed like an obvious conclusion in my mind. It’s just not happening. Forget that the first 3 years didn’t really have anything Security related. It was probably called-out, but nothing serious. The numbers show that the percentage of presentations (even the raw numbers) just aren’t increasing.

I wonder if similar conclusions could be made about Security in other conference settings and meet-ups associated with DevOps themes, as well as tooling to support the DevOps movement and Continuous Delivery. Is the problem with the Security community being slow to participate in the DevOps movement, or is it the DevOps community doesn’t know what to do with the Security community?

I honestly thought there would be more penetration of security within DevOps, as well as more penetration of DevOps within security. I had heard references to DevOpsSec. DevOps is real, but DevOpsSec isn’t there yet. The two are flirting and possibly going on a couple of dates, but it’s safe to say they aren’t going steady yet.

I talked about this with my colleague Jeff Williams. He eloquently described the situation between about Security and DevOps in the following sentence. “Security is still an island.” Jeff is right. People are talking about security in the context of DevOps and CD, but not in an emersed and integrated way.

I guess that’s what 2015 will all be about.

Why Am I Even Thinking About This

Let me start off by saying that this was not intended to be a controversial blog. Looking over now, it hopefully doesn’t read too controversial. I had just come from Gene Kim’s conference on Enterprise DevOps and I was starting to think more fluidly about DevOps, the Enterprise and Security. I was glad to see that some folks are talking about DevOps and Security.

There’s still not enough conversation. We need some more mind sharing in the space. We don’t necessarily need it all to come from Security experts. We need folks in the DevOps and CD spaces to jump into Security in a similar manner they did years ago with Performance and Scalability. It makes me think about my own passage to DevOps.

A few years back I was fortunate to watch one of my earliest mentors, Bernie Wong make the move from being a Performance Engineering practitioner to a Security Engineer. There are a lot of similarities between the two practice areas. Bernie convinced me that if I worked hard enough and embraced the Security community and all its good habits and flaws, then it would be an easy transition. I was fortunate to team up with a fantastic Security practitioner at Blackboard, Stephanie Tan to evolve an entire practice area around application security. I learned the ways of application security.

Stephanie and I worked on the problem of application security as part of an enterprise software product, as well as our SAAS cloud products for four years. We talked daily about how to make application security something “continuous” as part of our commit/build/test pipeline in which our tools could give our developers feedback within minutes of a commit. We also thought/worked the deploy and operations piece of live software, but not to the same degree as the commit pipeline.

We looked at every tool on the market. We built some tools as well and open sourced them. We committed to other projects via Pull Requests. We struggled to get a comprehensive tool set and workflow in place at a reasonable cost and without considerable security training. It’s safe to say that Security was an island. We had to run a parallel commit pipeline just of the AppSec team that couldn’t fail the build. We had a couple touch points with the developer commit pipeline, but not to the degree we wanted and visioned.

Alas, we both parted from Blackboard this past summer (amicably and on our own accord) to pursue deeper interests in the security community with other companies.

In full disclosure, I got the opportunity to work on the problem of Continuous Application Security for a company called Contrast Security. I’m not a co-founder, but I’m an early arrival. We evaluated Contrast about a year before and were really impressed. When they created the opportunity for me to join, I couldn’t refuse. It’s an opportunity to tackle Security and CD.

I welcome comments/feedback or tweets to anyone that wants to help make Continuous Application Security and the topic of Security within the DevOps community relevant. Also, the commentary in this blog are solely of mine and do not necessarily reflect the views of my current employer, Contrast Security or my past employer, Blackboard Inc. 

DevOps Enterprise 2014…A Conference Review of Sorts

I had the chance to attend a new conference on the DevOps circuit called DevOps Enterprise earlier in the month. For those of you who did not have a chance to attend, it was a conference co-hosted by one of the greatest Tech Connectors to have ever walked this Earth, Gene Kim, his colleagues at IT Revolution Press and the main sponsor Electric-Cloud. The timing of the conference was spectacular as it happened to fall around the five year anniversary of the DevOps movement.

https://www.flickr.com/photos/bethaniehines/15478309207/in/set-72157648631801518

Source: Flickr

I first learned about the Conference from a friend and colleague in the Washington, DC area named Jeff Gallimore, a longtime friend of Gene Kim and a collaborator on theDevOps Defense Audit Toolkit. Jeff had a speaking role at the conference with a few auditors and technologists, acting as the moderator on a panel about the Toolkit. Simon Storm, Josh Corman and Byron Miller participated in the panel and have been part of the Toolkit project for quite some time. I won’t go into too much about the Toolkit, other than to suggest reading the blog post and joining the Google Group if you have an interest in participating.

Pre-Conference Perspective

When I first thought about going to conference I was a little weary of what the topics would be like and who the attendees would be. The thought of DevOps concepts and culture inside the enterprise wasn’t something I naturally aligned. As a long-time Velocity Conference attendee, I was concerned that this conference would miss the mark and not teach me something new about the movement or introduce me to new ideas and practices. I personally watched the DevOps movement come out of Velocity, long before the term DevOps was even coined, so any other conference that was going to use DevOps as a culture would have to be top notch. Since Gene was running it, I gave him the benefit of the doubt.

Then I thought about my new role as VP of Engineering at Contrast, specifically about who are our customers. The bulk of our customers are not Silicon Valley startups or technology plays. Rather, they are enterprise players. Naturally, I thought it would be a good idea to see how the “Enterprise” was thinking about the DevOps movement and how they associated it with building and operating software system.

Thoughts on the Conference

I wrote a ton of crazy notes about each of the sessions from Day 1. I guess you can call me the “King of Horses” then. There was one session that stood out in my mind more than anyone and that’s the Target presentation. Ross Clanton (Ops) and Heather Mickman (Dev) presented about how the DevOps culture was brought into the organization at scale. It was how Dev + Ops found a home together at a large Enterprise player with years of legacy code and systems. You can also see the slides here.

Target is/was/will always be a big horse. They are doing the things that make them unicorn like to their competitors and their colleagues in other companies. What are those things you might ask? Well, first they are breeding a culture of transparency through a few core means. The first is making the move to collaborative development via Git using Pull Requests. Everyone uses Git on both the Dev and Ops side. They share repos and access across teams. Second, not only are they big players iMinneapolis’ DevOps Days, but they are running their own internal DevOps Days so that everyone in the organization can participate. Third, they are sharing to the outside world little by little via their Github blog.

The most impressive thing I waked away from the talk was this notion of Flash Builds. As they described, Flash Builds =
(flash mob + scrum + hackathon = awesome) 8hr day with 2x4hr sprints (includes retrospectives, planning, etc…). They promised a blog about the idea. Since it’s on Twitter and the Internet it has to happen ;). From their description, it sounds like Flash Builds are mini-hackday events that accomplish a full development and operations lifecycle. What this team showed as well as anyone is the need to breakdown silos and become one culture.

Devops More than Tooling

One of the pleasant surprises of the conference was the alignment of DevOps and Continuous Integration. I have to admit that I was totally expecting a ton of talks talking about the token DevOps tool sets for automation (Chef, Puppet, Ansible, Salt, etc…). I was expecting a ton of Docker, Vagrant, AWS Services and of course monitoring/reporting (Graphite, StatsD, CollectD, New Relic, etc…). The tooling was called-out in a few presentations, but it wasn’t three days of “Tool Overkill” and presentations about those tools.

I definitely felt like the enterprise players put a great amount of emphasis on build pipelines, feedback loops to development teams and collaborative deployment architectures involving development and operations working together. There were more references to Jenkins-CI than any other tool, followed by Git. In my mind that’s telling me the crowd was focusing on the development delivery pipeline striving for continuous development and feedback.

Unicorns, Horses and Ninjas

One of the main themes of the conference is the notion of horses and unicorns. The companies and players associated with the early DevOps movement were/are often referred to as Unicorns, as their work is considered magical and mystical. This has been echoed in papers, blogs and countless slides at DevOps Days, Velocity, FlowCon, OSCon and other conferences. It was fitting that Gene used this theme as I believe he calls out Project Unicorn in The Phoenix Project, a must read for any and all.

During the conference I found myself writing down some notes on what I thought a Unicorn looked like in the movement, as well as a Horse so I could put a picture to it. I even added a third persona called a DevOps Ninja. I’ll briefly post my comments on the three below. Note, I’m not intending to be snarky, but it reads pretty snarky.

Unicorns are companies that simply get it. They have broken down the culture barriers. There are no silos. Everyone practices Continuous Delivery, They post all of their awesome tools that they built in-house on Github. When they don’t build their own tools, they make time for Pull Requests on the latest and greatest projects on Github. They attend all of the latest conferences as presenters and sometimes as sponsors. Their most favorite part about conferences is the “Hallway Track”. They never sleep. It’s almost like they are vampire unicorns or something. If they aren’t coding on a plane, attending a conference or participating in a hack-event for a declining population of siamese albino wales, then they are guest speakers on podcasts. When do they have time to sleep or better yet…play golf?

https://drawception.com/pub/panels/2012/5-6/D59KOb3YZ7-10.png

What exactly are horses then? I guess it’s fair to say that they come on to the movement well after the movement is gaining steam, but rather has no become a fad. Why? Because they have been heads down on some important projects for the last few years and haven’t had time to pick their heads up. They manage their own source code (SVN, TFS or Perforce). They buy enterprise monitoring tools from IBM, HP and CA. They run their own data centers. They choose Perl and Shell as their scripting languages of choice. They go to conferences, but tirelessly take notes and even pictures of the slides on their smartphones and tablets. They don’t like to get up from their seats at the conference in fear of losing their spot.

http://image13.spreadshirt.com/image-server/v1/compositions/1001659506/views/1,,,appearanceId=231/horse---nerd-Women-s-T-Shirts.jpg

I decided to add a 3rd persona in the mix. Let’s call them DevOps Ninjas. Who are these ninjas we speak of? Well, they are the guys from Docker. Ah…just kidding. Well they might be from Docker, but essentially these are the guys and gals who skip the conference all together and have an “Un-Conference” or sponsor secret meetups. They are full-stack engineers who code-automate-deploy-fix-redeploy. They work purely in the cloud. They drift from Starbucks to Starbucks daily.

http://kyleart.com/wp-content/uploads/2008/07/07_ninjastarbucks.jpg

Should We Aspire to be Unicorns or a DevOps Ninjas?

I think it’s fair to say that there are no such things as unicorns or ninjas. They really don’t exist. I know a lot of folks who work for companies that are considered both and between you, me and this blog post they all have issues. Don’t get me wrong…they love their jobs. They feel empowered to make decisions. They collaborate a ton. They don’t live in bliss. Heck, watch a episode of My Little Pony and you will see that even the unicorn ponies on that show have drama. (Note: I am the father of two daughters ages 8 and 5, so we have My Little Pony on a queue at our house).

http://www.hdwallpapersos.com/wp-content/uploads/2014/08/My-Little-Pony-Wallpaper-Photos.png

I guess it’s fair to say that all companies can be compared to horses. You have ponies, jumpers, thoroughbreds, clydesdales, etc…Companies can be any or all at the same time depending on their attitude and culture, probably the best takeaway from the conference in my opinion. It makes me think of a great post by John Willis back in 2010 called What DevOps Means to Me. In the post, Willis references C.A.M.S. (Culture…Automation…Monitoring…Sharing), which many in the community identify as the four pillars of DevOps. I think the conference presenters nailed the C and S. Nearly every presentation focused on Culture and Sharing. Throw in the word Collaboration for C.S.C and you have a more robust picture of the mood and attitude of the presentations. Collaboration was truly being called-out between the traditional roles of Dev and Ops.

I think the Enterprise represents the line of demarcation between Development and Operations so well. For years you often had the development folks working in cubicles in the bowels of the corporate headquarters. The Operations folks they worked in some NOC or Data Center miles, sometimes many states away. Most of the time the developers didn’t know the operations folks and the operations folks didn’t know the developers. What the conference highlighted and the DevOps movement highlights so well is that the barriers have to be removed. The two cultures have to become an amalgam of one culture. These groups need to be together in order to breed a new ideology of thoughts and perspectives together.

Would I Go Back in 2015?

Absolutely…but I want to go back as a presenter. I felt like there were gaps in the stories I heard. The biggest gap is that Performance and Security continue to be called-out as something important to the Continuous Delivery pipeline, but it’s only lip service in my mind. The only folks who are making strides with Performance and Security remain to be the Operations folks. Incorporating Performance and Security as part of the build pipeline is not happening with the diligence and intensity that it should. These are by far the most important non-functional requirements of any system or application. Yet, they remain to be treated as second class problems by development teams.

The Power of Rundeck

A big part of the DevOps movement is the passion and commitment to “automate everything” and provide as much self-service as humanly possible. I’m a big believer in automation…not because I’m lazy, but rather because I have a desire to make all things repeatable, reliable and robust. I call those the “Three R’s” and they are a huge part of why I became a big believer in a 4th “R” which is called Rundeck. Note, I’m not the author of Rundeck. The awesome guys at SimplifyOps were the authors. I’m just a user, fan and admirer of cool, easy to use technology. Below is a quick passage about Rundeck…

Rundeck is an open-source software Job scheduler and Run Book Automation system for automating routine processes across development and production environments. It combines task scheduling, multi-node command execution, workflow orchestration and logs everything that happens. Access control policy governs who executes actions across nodes via the configured “node executor” (default for unix uses SSH) and does not require any additional remote software.[1] to be installed on them. Jobs and plugins can be written in scripting languages or Java. The workflow system can be extended by creating custom step plugins to interface external tools and services.

Wikipedia Comparison of Open Source Automation Tools

I’m a big fan of Rundeck for a number of reasons. My first reason is pretty straightforward. Basically, it’s a simple web application that provides the basic controls and workflow for self-service. The simple web gui is just so easy to use that anyone can understand how to use it with little training. My second reason is that it pretty much can make use of any automation/scripting framework out on the market. Third, it gives developers, operations engineers or even support staff a simple workflow for doing work on a server without ever logging into the server. Fourth and certainly not last is that it provides an audit and tracking system. There are other key things such as scheduling and reporting, which are super easy-to-use features to enjoy as well.

Source: Rundeck.org

Long before the SimplifyOps guys built Rundeck, my old team at Blackboard built an automation engine we called Galileo. My team built it in Groovy/Grails. It was a lot like Rundeck, but not as simple to contribute and extend. It served a great purpose during its time. It helped us achieve so many of the needs I listed above. It required a listener on each destination client. Rundeck works without an installation on the client system. All that’s needed is an SSH key or simply passing login credentials for a trusted user within the script.

Crowd-Sourcing Development

One of the cool things that the SimplifyOps guys do is crowd-source their development via Trello, which is one of the best kanban boards available (for freemium btw) on the market. Their board is public for everyone to follow, vote and even contribute.