Monthly Archives: March 2013

A Tester or an Engineer…It Shouldn’t Be an Either/Or

It’s 10:30pm on a Monday night. I just got off a skype call with a candidate for a position on one of my teams. The position is for a performance engineer. If you know me, it’s rare for me to hire someone on my performance team that has spent time as a performance engineer in the past. Maybe I’m a little biased, but nearly every candidate I’ve run across over the years that’s been a so-called performance engineer was really a performance tester in wolves clothing. 

Maybe I shouldn’t be so tough on other performance engineers. This candidate was another wolf in sheep’s clothing. His resume said performance engineer, but it read performance tester. I should have gone with my instinct and passed on the candidate.

So why am I being so hard on other performance engineers who really are performance testers? It’s not like I don’t see a place in the software and technology world. I do…I just have higher expectations. I think you should too…

I like my performance engineers to have a passion for reverse engineering. I want them to logically explain the execution of code from top-to-bottom. They should be able to explain the efficiency of one method call, routine, execution plan, etc…over another. They should have a passion for forensics and not just the show CSI. Forensics from a software performance perspective is about using quantitative approaches to measuring performance and scalability. It’s about balancing “Fuzzy Forensics” with more “Transparent Forensics”.

By “Fuzzy Forensics” I’m referring to data that can be used in an abstract fashion for correlation purposes. An example of a “Fuzzy Forensic” piece of data might be the output of a performance counter against the CPU or the GC output from a JVM log. The data can help narrow down the window of time, a resource constraint, a hotspot in the architecture or a bottleneck, but it can’t provide the depth of explanation to achieve root cause on a performance issue. That’s where “Transparent Forensics” comes into play…A good example of this might be Call Stack Tree, Heap Dump, SQL Trace, etc…The cost to obtain “Fuzzy Forensic” data is far less expensive than “Transparent Forensic” data. 

Back to my point…a lot of folks who call themselves performance engineers aren’t even thinking about forensics. The thought of measuring performance is foreign. I keep hearing this horrible broken record that measuring is done by the tool, whether that be LoadRunner, JMeter, SilkPerformer, SOASTA, Grinder, etc…That’s simply not the case. 

So why do I care? Here’s the thing…in all my years as developer and system engineer, I’ve been passionate about performance. I’ve been a practitioner of performance engineering for nearly my entire professional career. I perk up when I meet other performance engineers. Every now and then I get really frustrated when I meet someone who enjoys the various aspects of performance engineering, but doesn’t take it further. If anything, it’s a greater frustration with humanity for not being willing to challenge themselves to be omniscient in their field. 

It makes me think of this analogy of sorts. A bunch of kids are part of a high school track and field team. It’s a new program to the school and none have any prior experience with the sport. The school doesn’t have a lot of money so the kids mainly practice their events on a grassy field behind the school. The kids have a lot of success practicing. Some of the kids are separating themselves as the truly talented team members. While others are simply improving day by day with personal bests in their events.

After several weeks of steady improvement they go to their first meet against a series of local teams. It’s an invitational of sorts. All of their meets are away as they have no facilities to host a meet.

The kids show up to their first meet and step on the track at the stadium. Nearly all of the kids are in awe. They hadn’t seen a track so nice. Remember they were used to practicing on a grass field behind the school. The first race was a total disaster. Only 1 person scored a single point, the one shot-putter. 

The kids who were considered fast by their teammates, finished last behind every other runner from the other schools. None were prepared for the terrain. None were prepared for the technology. The kids left the meet pretty discouraged.

The next day in practice, one of the kids suggested that they consider running on the pavement instead of the grass. Another team member suggested that they go to another public school on the weekends to practice. Another member suggested that they change their uniforms from t-shirts, shorts and jogging shoes to more aerodynamic outfits.

These were the kids that picked up on the visual clues around them at the meet. Some of the kids didn’t buy the hype. They wanted to continue practicing on the grassy field. They wanted to  wear their normal uniforms. They didn’t want to practice on the weekends. These were the kids who didn’t grow all that much.

The next meet yielded better results. Rather than being embarrassed with only 1 point, the team scored 20 points. They even had a contestant finish in the top-2 for their event. The talent wasn’t any worst. In fact, it was the same teams from the previous meet, just at a different location. The kids who picked up on the visual clues were the contributors to the team’s success. 

The difference from 1 point to 20 points was learning was happening. The kids who were scoring points understood that the environmental conditions they practiced against were very different from the environmental conditions they raced. They had to be both visually aware and perceptively aware. Essentially they had to do more than just adapt. They had to evolve. They had to be able to reason. They had to survive. They had to trust their own instincts and observations. Most importantly they had to change.

I think this is one of the greatest threats facing performance engineers today. The good performance engineers realize that they have to constantly learn. They have to challenge their process, not in a confrontational manner, but in a socratic manner. 

The really good performance engineers get better with each day.


Book Review of the Phoenix Project

Back in February I wrote a blog about how Amazon recommended the Phoenix Project about five minutes before I walked into Mike McGarr‘s office only to realize he was reading the book himself. I ended up reading the book, now twice cover-to-cover and I’m ready to share my thoughts…

I don’t know if I’ve ever really given a book review in the past. I know over the year’s I’ve recommended books like Seth Godin’s The Dip, Malcolm Gladwell’s Outliers and Blink, as well as Chip and Dan Heath’s Made to Stick or any of the Patrick Lencioni books, though my favorite was Three Signs of a Miserable Job. What I’ve come to realize is that I like books that are relevant to career, survival, instinct, etc…that are presented as stories. The stories can take different formats. Patrick Lencioni and Gene Kim (Phoenix) tell their stories as fictional narratives with underlying business messages. Gladwell, Heath and Godin tell non-fictional, mini-stories about real people to relay their points. I’ve come to conclude I like both styles. I’ve also come to conclude that I don’t like books that read like college text books. None of the books above read like a boring text book 

Why You Should Read This Book

Let’s cut to the chase…I think you should read this book because it’s a riveting story that 99.99999% of all people not living under a rock can relate to. You don’t have to be in technology, manufacturing, business, etc…to find a kinship with the characters and storyline of the book. If you are reading my blog, most likely you are some form of technologist, whether that be a software developer or engineer. So this story will resonate like no other. You will find yourself associating character’s with real people from your own life experiences. I found myself chuckling over and over as I realized that I experienced or am experiencing (present day) many of the same challenges that the protagonist, Bill Morris and his team were facing.

I also took a lot of notes and performed a lot of reflection. I quickly realized that over time I managed to become quite complacent with my own work and the work of my team. I had a lot of in-flight work. I had no controls in place with regards to accepting work. Priorities were managed by who screamed the loudest. I kept asking myself, how in the world did we ever get any work done?

What I Learned from the Book

There are a ton of underlying themes and messages in the book. I took a few key points away from the book. These points ranged from knowing the origins of my work, controls of work being accepted or rejected, the flight/progress of work, prioritization of work, continuous improvement (kaizen) and the relevance of practice of work (kata). I’ll cover the three most important below.

Origins of Work
Work needs to be better categorized from where it’s coming. Work that comes from a customer or key stakeholder has different controls than let’s say a self-created initiative. The author’s call-out four types of work: Business, IT Operations, Changes and Unplanned work. It’s that fourth type of work (unplanned) which is the silent killer to a team. In my own work scenario, I’ve narrowed down work to three types: External (Client), Internal (infrastructure) and Unplanned.

Controls of Work
I’ve never gone so far to create a process for work to be accepted or rejected. I’ve definitely rejected work in the past, but never a formal process. What I didn’t realize that by not having a formal process for accepting work, it creates a high-degree of chaos in terms of a backlog of work. The stuff just keeps piling up until it gets done. When work piles up, priorities get mucked.

Work in Progress
The biggest take-away thus far for me has been an interest in learning about Kanban. All of my teams are using a Kanban tool, essentially a board that visualizes work, queues up tasks and acts as a pull-based approach in which we pull work in to the group. This comes as a result of the controls of work being put in place. We can visualize our actual WIP (Work in Progress). We have awareness of unplanned work (which takes a long time as it requires 100% transparency). We can help each other prioritize the work of the team.

What I’m Reading Next

I have two books that I’m actively reading. The first is The Goal by Eli Goldratt which is eerily similar in genre to this book. Goldratt’s Theory of Constraints is referenced constantly throughout the book. The second is Kanban: Successful Evolutionary Change for Your Technology Business which is more of a HowTo on Kanban.

Good Days to Start and Stop a Sprint

So let me start off by saying I’m learning a lot about agile development practices which I’ve never paid a lot of attention to. I’ll be the first to admit that I’ve never gone to any Agile training. I’ve read 3 books, a handful of articles and listened to a ton of lectures or YouTube videos online. I haven’t been entirely committed to Agile. I’m not quite sure why because I know I don’t have any objections to it.

Since my role changed back in the summer and I have a couple of new teams, namely the DevOps and DBA teams, I’ve had to think a lot more about getting more productivity out of all of my teams (Performance, Security and BbLabs included). Watching Nori’s approach to distributed teams with Trello, Stephanie’s quick adoption soon after and Mike’s SCRUMs, Retrospectives and physical Trello board in the DevOps room, I have to say their activities have really opened my eyes to the benefits of adopting more Agile practices. Here’s a few observations…

  • Let’s our teams be less command and control…to being more self-managing
  • We have a place to track and reference “Unplanned Work”
  • We can call-out and jointly prioritize initiatives for the team

All of this Agile talk has got me thinking about sprints, stand-ups and agile boards. I thought back to when I was running sprints for the DevOps team while I was recruiting Mike. I had a weekly meeting, usually on Monday’s which sucked. It was my call to only meet formally with the team 1 day a week and it was usually on the first day of the week. There were a couple of problems with that Monday schedule that I never really took the time to think about changing. Monday also happened to be the day we started sprint cycles and the weekly beginning of each sprint week. We ended all of our sprints on a Friday.

We had a holiday on one of those Mondays and if I recall we had a ton of vacation days on Fridays after the company announced no carry-over going forward. It was definitely an issue from my perspective to have days we started sprints or sprint weeks, as well as days we ended sprints or sprint weeks with holidays or missing people.

So what am I getting at? I’m just throwing out the idea if we started sprints during a Tuesday, Wednesday or Thursday, would the sprint behavior change? Would a weekend gap between sprint weeks actually be positive or negative?

Apparently I’m not the only person asking this question…


This is Scary What Amazon Just Recommended to Me

So I just wandered into Mike McGarr‘s office earlier today. He was on the phone so we couldn’t talk. If you ever walked into Mike’s office, you would know that he always leaves his current NY Times Best Seller on his desk. Today’s new book on his desk is The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win. Mike’s words were pretty clear and simple…”You have to read this because John Allspaw likes this book. In fact, he wrote a blog about how awesome this book is.”

So naturally because Mike likes it and because Allspaw likes it…I just had to go search it on Amazon.

This is where the story gets better. Amazon must have implanted a secret chip in my head. Before I could even search for the book…it’s was in the #1 spot on my list of recommendations. Very tricky Bezos…way to be spooky and awesome in the same breath!

Making the Move to Evernote

I’m a writer and I love my notebook. In fact, I’ve been using the same notebook for over 10 years. I first found this notebook at Barnes and Noble. It’s a multi-colored and lined. I think over the years, I’ve created 1 to 2 per year. Looking over at my bookshelf, I’m count 14 on the shelf plus 2 on my desk. I am pretty sure I have a couple at home that never came to Bb from my previous job.

In fact, this past week before I went to the Sales Conference in Austin, I quickly ran over to Barnes and Noble to buy a brand new one as my journal I used for most of 2012 had less than 2 pages. So now I have this new notebook. It looks great and it has lots of unused pages…

Most likely it’s going to continue to have unused pages. I’ve decided it’s time to make the jump to the digital world. After a little bit of balking, I’ve decided to start using Evernote. I’m in my second day of using it, mainly on my Mac and then checking notes on my IPhone.

I think I am going to take the next steps and breakdown and buy an IPad, either the Retina Display or the Mini. The only way I will do that is if I end up buying the keyboard attachment.