The Sixth Sense of Benchmarking

A few days ago I was having a conversation with some folks about the unlikeliness of a movie like the Sixth Sense or even the Usual Suspects working in today’s connected society. People would twitter, post to facebook or blog the outcomes of those movies within minutes. They would be leaked over YouTube or covered in US Weekly. It’s kind of bummer because those were two awesome movies. From my point of view, they worked and I would love to see more movies like them in the future.

This got me thinking about what it means to have a Sixth Sense in the world of benchmarking. Is there some form of clairvoyance that performance engineers should have in general? I think so, which is why I’m writing this blog. Performance engineers should constantly be thinking about the line that exists between the software execution model and the system execution model. The software execution model characterizes the resource requirements of the software alone, without considering other workloads, multiple users, or delays due to resource contention. The system execution model characterizes the performance of the software in the presence of factors which could cause contention for resources.

I tend to think about the software execution model in terms of focus tests and isolated design of experiments. I think of the system execution model in terms of design of experiment, but mainly from the perspective of system components. The system execution model is what benchmarks are all about. So what exactly is this Sixth Sense I reference, but don’t actually define?

Benchmarking is about telling stories. Any given benchmark might contain 1, 2, 3 or even 10 stories. The more interconnected the stories, the better the benchmark. Performance engineers have to be able to seek out the story lines, find the connections between the stories and figure out a way to present combined stories as one big story. So the Sixth Sense is the ability to tell a really good benchmarking story.

Our team just came back from our most successful benchmark ever in the history of our team. We had a number of story lines around platform technologies, as well as a series of stories with regards to behavior and adoption of our system. We wanted to study the system execution model for an institution primarily using Blackboard as a distance learning tool. That meant a 100% online interaction model for everything from course delivery to collaboration to assessment. Sessions would be longer. Clicks would be deeper. Most importantly a larger percentage of users would be concurrently active in the system at any given time.

We had a lot of technology story lines to study as well. One story was around identifying the best set of standard (-X) and non-standard (-XX) JVM options that would reduce our latency introduced at the application layer. There was a subplot about JVM sizes in a 64-bit stack. Was there a big difference between a 2GB vs. 4GB vs. 8GB vs. 16GB JVM? Another story was constructed about memory management in Oracle 11G. Of course we had a great story line about virtualization using Citrix XenServer with regards to identifying our optimal collapse ratios.

There are a lot of good stories available for us with our current release. I’ve seen one good story thus far in which the team would like to compare the performance and scalability of virtualization versus clustering as a deployment strategy. I would love to see a story about SQL Server 2008 from a scalability perspective. A lot has been written about performance and scalability of the platform. There’s also a webcast worth listening. We should have dozens of questions about the latest technology. Though I’m partial to always investigating the query optimizer as my first initiative when it comes to an updated database.

These are just a few of the questions we have to ask as we prepare benchmarks going forward. Coming up with questions is the best way to build out plots which ultimately make our stories.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s