It’s 10:30pm on a Monday night. I just got off a skype call with a candidate for a position on one of my teams. The position is for a performance engineer. If you know me, it’s rare for me to hire someone on my performance team that has spent time as a performance engineer in the past. Maybe I’m a little biased, but nearly every candidate I’ve run across over the years that’s been a so-called performance engineer was really a performance tester in wolves clothing.
Maybe I shouldn’t be so tough on other performance engineers. This candidate was another wolf in sheep’s clothing. His resume said performance engineer, but it read performance tester. I should have gone with my instinct and passed on the candidate.
So why am I being so hard on other performance engineers who really are performance testers? It’s not like I don’t see a place in the software and technology world. I do…I just have higher expectations. I think you should too…
I like my performance engineers to have a passion for reverse engineering. I want them to logically explain the execution of code from top-to-bottom. They should be able to explain the efficiency of one method call, routine, execution plan, etc…over another. They should have a passion for forensics and not just the show CSI. Forensics from a software performance perspective is about using quantitative approaches to measuring performance and scalability. It’s about balancing “Fuzzy Forensics” with more “Transparent Forensics”.
By “Fuzzy Forensics” I’m referring to data that can be used in an abstract fashion for correlation purposes. An example of a “Fuzzy Forensic” piece of data might be the output of a performance counter against the CPU or the GC output from a JVM log. The data can help narrow down the window of time, a resource constraint, a hotspot in the architecture or a bottleneck, but it can’t provide the depth of explanation to achieve root cause on a performance issue. That’s where “Transparent Forensics” comes into play…A good example of this might be Call Stack Tree, Heap Dump, SQL Trace, etc…The cost to obtain “Fuzzy Forensic” data is far less expensive than “Transparent Forensic” data.
Back to my point…a lot of folks who call themselves performance engineers aren’t even thinking about forensics. The thought of measuring performance is foreign. I keep hearing this horrible broken record that measuring is done by the tool, whether that be LoadRunner, JMeter, SilkPerformer, SOASTA, Grinder, etc…That’s simply not the case.
So why do I care? Here’s the thing…in all my years as developer and system engineer, I’ve been passionate about performance. I’ve been a practitioner of performance engineering for nearly my entire professional career. I perk up when I meet other performance engineers. Every now and then I get really frustrated when I meet someone who enjoys the various aspects of performance engineering, but doesn’t take it further. If anything, it’s a greater frustration with humanity for not being willing to challenge themselves to be omniscient in their field.
It makes me think of this analogy of sorts. A bunch of kids are part of a high school track and field team. It’s a new program to the school and none have any prior experience with the sport. The school doesn’t have a lot of money so the kids mainly practice their events on a grassy field behind the school. The kids have a lot of success practicing. Some of the kids are separating themselves as the truly talented team members. While others are simply improving day by day with personal bests in their events.
After several weeks of steady improvement they go to their first meet against a series of local teams. It’s an invitational of sorts. All of their meets are away as they have no facilities to host a meet.
The kids show up to their first meet and step on the track at the stadium. Nearly all of the kids are in awe. They hadn’t seen a track so nice. Remember they were used to practicing on a grass field behind the school. The first race was a total disaster. Only 1 person scored a single point, the one shot-putter.
The kids who were considered fast by their teammates, finished last behind every other runner from the other schools. None were prepared for the terrain. None were prepared for the technology. The kids left the meet pretty discouraged.
The next day in practice, one of the kids suggested that they consider running on the pavement instead of the grass. Another team member suggested that they go to another public school on the weekends to practice. Another member suggested that they change their uniforms from t-shirts, shorts and jogging shoes to more aerodynamic outfits.
These were the kids that picked up on the visual clues around them at the meet. Some of the kids didn’t buy the hype. They wanted to continue practicing on the grassy field. They wanted to wear their normal uniforms. They didn’t want to practice on the weekends. These were the kids who didn’t grow all that much.
The next meet yielded better results. Rather than being embarrassed with only 1 point, the team scored 20 points. They even had a contestant finish in the top-2 for their event. The talent wasn’t any worst. In fact, it was the same teams from the previous meet, just at a different location. The kids who picked up on the visual clues were the contributors to the team’s success.
The difference from 1 point to 20 points was learning was happening. The kids who were scoring points understood that the environmental conditions they practiced against were very different from the environmental conditions they raced. They had to be both visually aware and perceptively aware. Essentially they had to do more than just adapt. They had to evolve. They had to be able to reason. They had to survive. They had to trust their own instincts and observations. Most importantly they had to change.
I think this is one of the greatest threats facing performance engineers today. The good performance engineers realize that they have to constantly learn. They have to challenge their process, not in a confrontational manner, but in a socratic manner.
The really good performance engineers get better with each day.