Tag Archives: forensics

Our Performance Core

We do a lot of things in Performance Engineering. Some good…some not so good. Because we do so many things, at times we get distracted. Our ability to progress is hampered by our inability to remove work from our plate. We may not even want to remove the work. The work for some is so appealing that at times it becomes more interesting than the work that might be better positioned for our team. I digress…

As we finish up 2012, the 9th full year of performance engineering here at Blackboard, we have the opportunity to get back and focused on our core with a little more rigor and a little more direction. I wanted to use this blog as a starting thread about our core. We need to re-establish our identity and remember who we are and what our team needs to be. That’s a little different than the notion of who the individual wants to be. Our team is the performance team. Now individually, many of us have passions and goals beyond performance. That’s great…heck it’s encouraged. We have to be true to our core and realize that our personal wants can’t come at the sacrifice of our core.

I’ve simplified our practice areas to four main services: Performance Design, Customer/Production Forensics, Performance Development and Benchmarking/Testing. I will cover each core area in the section below. All four of these are simply components of a larger enterprise. That enterprise I’m speaking about is Continuous Delivery. We all (beyond Performance Engineering) have to get on the same page that we need to be connected to all of the teams that make, support and operate this product.

Core Service Brief Description
Performance Design We often refer to this as a team as SPE (Software Performance Engineering). I think we need to stop, because it’s a mistake to call this as SPE. SPE is more encompassing and crosses into other service areas. The heart of this service is to build product for performance and scalability. We collaborate with development from customer need, through requirements, next to design and then through development. We model, assess risk, build test artifacts, evaluate code (static and dynamic), micro-benchmark, refactor and start the cycle all over before releasing to system test. We need to view this service as a service that happens before deployment.
Production Forensics We have been doing Tier-3 escalation for years and have been doing it well. We need to continue to improve in this area. There are two main areas I think we can improve. The first and obvious is greater emphasis on root cause analysis. What was the cause of the performance defect and why was it introduced. The second is the missing piece to our success, which is the how can we solve the problem of why it was missed during performance design or even during system benchmarking/testing? That leads to a second effort which involves figuring out whether we are exposed to future risk because we are missing something.
Performance Development We really have two development initiatives on this team. We have to prioritize based on need. The most prominent from a need perspective is our maintenance development, specifically for patches and defects that come in as diagnosed defects. We need to be on top of our game for this work. We cannot have mistakes and our degree of confidence from this work must be high at all times. We must put in the most amount of testing, profiling and micro-benchmarking as possible to make sure this work resolves our customer’s issues. The second is prototype development for the sake of performance. This is new feature or platform development for the sake of performance and/or scalability. We have to be 100% confident that what we are developing makes our product better and does not expose our product to greater risks.
Benchmarking and Testing This is our bread and butter. To many in the organization, this is one of the most important if not the most important exercise we perform as a group. This is focused on incorporating benchmarking and testing into our Continuous Delivery Pipeline (see image above). We need to fit in multiple places in the pipeline. It’s not 1 place. It’s in multiple places, providing both developer feedback loops, as well as system deployment/configuration guidance.

Slides from BbWorld 09 Main Conference Forensics Presentation

Hi readers,

Attached are links to the forensics presentation and a sample neXpert report. Thanks for everyone who came to the session. For those who couldn’t because of the Sys Admin group meeting, well you missed a good one 😉

Enjoy the slides!

1. Main Presentation

2. neXpert Report