Hiring Engineers: Why We Do Open Book Code Challenges

One of the few topics my friends and colleagues in the industry are willing to talk about are their strategies for recruiting and hiring talented engineers. It’s kind of funny because as an industry of practice, software engineers are so open about everything. Recruiting and hiring is such a competitive exercise, it feels like only the big giants like Google, Facebook and Apple go out of their way to talk about their approaches.

For this blog, I am going to talk about the topic of interviewing candidates. Like most companies, the process takes time and effort. Most candidates often make upwards of two visits to our office. Our average time to go through the process of introduction to offer takes about 4 weeks. In the tech world, that’s like the equivalent of two decades.

It’s Not Just a Job

When I got out of college in the late 90’s, people would jump jobs every 4 to 8 months. If your resume wasn’t littered with startup after startup, or enterprise after enterprise, you weren’t really playing the game right. For a young pup like myself, who watched his father have the same job for 50 years running his own dental practice, it was mind blowing to think the industry approved of this rampant mobility.

I approach things from the perspective that a job isn’t just a job. Rather, a job is an experience in your life. It becomes part of your DNA, just like going to high school or college. Your job consumes 1/3 or more of your time each week, so you better make sure it’s the right place.

We want our candidates to want to be at Contrast. I’m sure everybody says that and wants to believe that phrase to be the gospel. We truly believe it. We want Contrast to be your startup…your top choice. We’ve passed on some really talented engineers that didn’t see much value in our process.

Meeting the Team

I read an article a few weeks back about Etsy which referenced the word “culture fit”. It made me chuckle to say the least. We try our hardest to build a team of skilled engineers who want to work in an environment that values respect for one another and offers the chance to speak open and honestly. We don’t look for clones or brogrammers. We look for engineers that want to make the Internet safer and want to be a part of a high-performing team.

I think it’s important to sell my team from the start. I want every candidate, whether their an intern or a full-time engineer to trek to our office for a meet and greet. The meeting can take anywhere from 30 to 60 minutes. While it consumes a good amount of my time and pulls my engineers away from their assignments, it’s the first step in establishing a relationship of trust and openness with a candidate.

We are pretty transparent from the start. This initial meeting is not an interview for the candidate, but rather the chance for the candidate to interview us. We want our candidates to come into the office with a boat load of questions to make sure this is the right place to interview.

The Project

Every engineer that’s been hired at Contrast since I started has had to do a project. The project has changed over the years. The link above shows you the current state of the project. The project is open book and candidates do not have a time limit to complete. We often advise candidates to work on it over the course of a week or two at most, otherwise they won’t complete it.

The project is wholly owned by the author. We don’t want our names mentioned in the project. We do ask that the author codes the project and submits it to Github. Some folks are uneasy about a coding project in the public persona. Most like the project as it gives them a reason to build up their public coding profile.

We ask candidates to do projects for a couple of reasons. First, the project is designed to showcase a candidate’s ability to take on new work and see how serious they are about working with us. The really serious candidates crush through the assignment in less than two weeks. Second, the project is used as the focal point for the majority of the interview (see below). Third, the project gives us a chance to see how well you deal with unfamiliar technologies, as well as demonstrate your creativity.

The Interview

Most technical interviews are awkward, frustrating, unrealistic to name a few attributes. Our technical interview isn’t about how great of a developer you are. Rather, it’s about going through the experience of working a project and receiving/giving peer feedback.

We generally perform a small code review of 2 to 4 engineers with a candidate. We once had 12 people in a room with a single candidate, which we really blew. I won’t make that mistake again. Apologies to that candidate as the interview was way too overwhelming.

The code review starts with the candidate walking us through the project. He/she tells us how they interpreted the problem and how they decomposed their work. Next, they walk us through their design and implementation. They are then given the chance to call-out things they would do differently before we ask questions.

The interview team then asks upwards of 4 to 6 questions about the project, design choices, implementation decisions and further elaboration of what’s next. It’s not all questions though…my engineers will always add commentary of what they enjoyed or were concerned about with any project. Most of the time the candidate takes the feedback real well and comes back with a good follow-up question or response. Occasionally, we have a candidate that’s upset with the feedback, but can’t offer a good path forward. When this happens, both parties realize it’s not an ideal match.

The Experience

The overall experience of meeting the team twice and working on a project can be overwhelming to some who are looking for quick work. For the folks who’ve joined our team, it was a good formula for making the decision about whether this was the right place to work.

It may take time and in our space, time is not unlimited. It’s safe to say that it’s the right time for everyone to make an educated decision about their next job or next hire.

We’re Still At It…

Two years ago, I declared in my last blog entry that I would make “more time” for blogging and low and behold, I failed to do that. Shocker right? It makes sense that my readership dropped to practically none. I think my wife and a few folks for an Amazon Click Farm are my only loyal readership at this point.

I figured that I needed to write an “ice breaker” blog before I start blogging more. I do have more to write and more to say. So I will use this blog as my uncomfortable attempt at pretending I’ve been blogging all along.

The Recap

In my last blog, I teased my massive readership about our move to the historical Natty Boh Tower in Canton. Two years into it, I have to say, every day I love driving to the office and seeing/working with my team. We are a lot bigger than when we moved in back in the summer of 2015. We’ve added 10 more engineers to the office, with plans to add another 4 full-time engineers and 3 summer interns in a matter of weeks. As I mentioned before, the fine folks at Hyperspace outfitted the entire office with Steelcase and Turnstone. I will have to post some pictures in an upcoming blog.

When I started, we only had a handful of customers. I could namely them all and tell you the main point of contacts by their first name. For exactly two years, I wasn’t only the VP of Engineering, but I also served as our primary Tier-1 engineer. Now we have an entire team of support engineers and I’m still learning our newest customers as they on-board.

Initially when I started, we had (1) agent that was GA, our Java agent. We had a second agent that was technically in beta, our .Net agent. We were about to begin our third agent, NodeJS. We had practically no integrations or Open Source contributions.

Today, we are putting the finishing touches on our 4th language, Ruby and underway with our next language, which I will keep under wraps until we get closer to beta. We’ve got two modes to our language portfolio: Assess and Protect.

Probably our biggest news since I last wrote, outside of our growth (customers, people and product) would be the investment we received last year from our Series-B. When we took in our Series-A, we were really heads down and quite frankly all working 18-hour days and most weekends. I still work 18-hour days from time to time, but now it’s more a modest 14-hour day 🙂

Our Series-B lead investor is General Catalyst and we are fortunate to have Dr. Steve Herrod on our board. Steve is the former CTO of VMWare and quite frankly the mentor that I needed to challenge and motivate me.

What’s Exciting About the Now

The fun part of my job is that I don’t have to spend hours and hours pushing paper. My last 3 years at Blackboard felt like I worked perpetually in HR and Finance. What I loved about my first 8 years at Blackboard was how hands on I was with our technology. At Contrast, I love that I’m so involved in our development and deployments. It’s tough to juggle being an executive, while at the same time influencing our technology, but it’s worth the late nights and weekends.

We’ve got many parallel projects happening right now, but there are probably a couple of projects that really get my psyched at night to continue working. I’ll give the quick highlights of each project and why I’m enjoying them below.

Serverless (AWS Lambda)

I’ve been an eager beaver to work with Lamba for over a year now. I didn’t go to reInvent when it was announced a while back, but I certainly did my research after the announcement. As an application guy who is a firm believer in distributed program and agent-based pipeline architectures, the notion of serverless computing seemed mind-blowing.

My colleague, David Hafley, who joined me at Contrast after Blackboard, were both looking for a project or two to get started. We challenged our team to consider leveraging Lambda and forking a GoLang project called Goad to build an elastic benchmark kit for Contrast. Our plan is to release the kit to our customers and speak at a series of summer meet-ups to share the experience. This was only (1) Lambda project. We have a few about to kick-off.

What I love about this project is that we just don’t know how Serverless will land over the next couple of years. If it truly takes off, we have the opportunity to be a part of a crazy ride. The unknown is kind of fun, not only because it’s different, but because it’s unchartered.

Building a Language Agent…Fast…Really Fast

I had the option to hire this one particular engineer either as our Full Stack lead or to go build our Ruby agent with another team member. He and I came to the conclusion that building a Ruby agent would be more fun and beneficial to the company. We’ve built up the agent very rapidly. We’ve had a lot of learning with our first three agents to leverage, as well as a fair amount of shared code (you would be surprised). While we haven’t GA’d the agent yet, we do have the agent running in a few customers’ environment. They are having great success with the agent.

What’s cool is this is really the first agent that I was able to influence how we assembled the team and approached our development. I’m stepping out of my comfort zone in engineering and helping our marketing and sales teams prepare to bring this baby to market.