SIEGEL: The relationship between you, the artist, and the piece of music that you’ve performed over decades, is it something constant and steady, or are there moments of evolution when you decide after some performance, I’m going to do this differently now, I mean, I’m going to rethink that particular movement? Or does it just change very naturally without being so self-conscious about it?
BELL: Well, a lot of what happens in – and as a – we’re basically evolving and learning. You’re a constant student as a musician. So – and a lot of that, it just evolves naturally. But there are also moments where the light bulb goes off in your head and quantum jumps in understanding that those happen as well, and those are wonderful moments. You know, those often come after times where you feel a little bit stagnant or frustrated.

But I believe in things happening naturally. I think if you try to do something artificially or try to change your style or – very self-consciously, I think it won’t be honest.

SIEGEL: But even trying that hard, it’ll change because you’re growing as an artist.

BELL: It does change. We all – I mean, we change anyway. I mean, as human beings, we change. We – the more experience we have in life, it all informs and is reflected in everything we do artistically.

What got me thinking was Bell’s notion that we are constantly evolving and learning. We are “constant students” he says. Interesting enough as software engineers it’s rare that continue our education. The best programmers are learning new patterns, languages and skills. The vast majority of software engineers and software practitioners fail to continue their education, whether in a formal or informal manner.

When I got 2 or 3 months without learning something new, I feel stale. I hate that feeling. Lately I’ve been teaching myself Ruby and re-learning MySQL. I’ve been using the MCPS library and their online access to the O’Reilly catalog as my main resource. I don’t necessarily need a class or a workshop, just a little motivation.

A dream scenario would be that we as a community put our heads together and co-author a more collaborative document that’s living and breathing. It always feels as though once we put out a PDF, the document is immediately obsolete. Hopefully that won’t be the case with this guide. We will have to see.

We are in the early outlining stages of the document right now. I’m mainly putting together a master list of platform components to focus on. The most glaring holes as I see from the last guide is the incredible lack of information we left out on Blackboard specific information such as Snapshot, Content Exchange and Bb-Tasks to name a few. That’s my bad as I organized the last document and wrote most of the content. The other big hole that I’m seeing right now is Lucene. We have a few implementations of Lucene in the product that never gets the appropriate care and feeding.

My ultimate goal is to get the customer community involved in writing this blog. I did a quick Twitter post a few minutes ago soliciting some early thoughts. I’m hoping to get some feedback from the community about what they want in the guide, versus what would they remove. Now iss everyone’s chance to have their voice heard.

Send me a DM to @seven_seconds if you are interested in participating.

You can access the slides here:

There are some basic tenants of performance benchmarking and testing that can be followed to pull off this feat. It starts by setting goals, targets and thresholds which I will describe later. You will subsequently need to define both functional scenarios and data requirements for test interactions. No performance test is complete without writing test automation. Don’t fall hard for using “real users” to perform your tests. Why might you ask? Well a) it’s not scientific b) it’s not repeatable and c) it’s pretty tough to get hundreds of users to participate in a synchronized activity. Before a single test is run, you will need to plan how the scenarios will be distributed, sequenced and balanced, as well as the attributes of a test such as duration, workload, arrival/departure rates, think time, timeouts and abandonment.

Who’s to say it doesn’t have to happen. In fact, lots of great software companies and organizations actively seek out issues that go unreported. They build frameworks into their products to notify development and/or support teams that an issue occurred. While it might not be reported by the user, it’s actually being reported by the system. Now that would be a dream for me. Imagine an administrative feature set that sends some form of message when an exception occurs? Granted, we might get bombarded with hundreds or thousands of these messages, but we could easily build the capability to aggregate and analyze the data so that issues could be evaluated both individually and as a holistic trend.

Building an administrative feature wasn’t my intention of this blog. Rather, I wanted to talk about the topic of proactively evaluating issues that may or may not be reported by issues. I call this forensic fishing. The idea is to strategically sample a software system with the intent of identifying a performance, scalability or even a functional issue as part of the sampling effort. The goal is to seek out issues or catch some fish. It’s not like you go fishing with the intent of going home empty handed right?

So why do I use the phrase “strategically sample”? Well, my first strategic purpose would be to sample customers who are early adopters of a release. Let’s reward early adopters by responding to issues before customers report them. The second would be to sample a customer who’s taken a patch or SP based on a fix we did. Our team is working dozens of issues per release. We verify in our lab, but ideally being able to verify with an even greater degree of confidence. Third, the whole goal of fishing is to find issues. Issues creep from release to release and go unnoticed. We may find an old issue that gets exposed because our magnify glass was fortunate to pass over it during a sampling period.

Of course our plan is to fish using a pretty awesome rod, aka…Dynatrace. The idea is to build a program with our colleagues in Managed Hosting to selectively pick customers for sampling. We have to identify when we sample and most importantly how long we sample. I’m a firm believer that we do not need to sample every aspect of the customer. We could run with a lightweight profile. We could selectively choose 1 node or maybe 2. If a customer had 10 nodes, it’s not worth our time to sample them all. We obviously need some low hanging fruit to initially focus on and react to. A more sophisticated inspection could and should be performed as well. What would we be looking for? We would look for everything from exceptions to unreleased connections to other forms of software anti-patterns. We would sample over short intervals like 30 minutes and use the compare features across profiles.

So what does that have to do with Blackboard Learn?

Good question Steve. Let’s start by talking about Exceptions and Log Messages. I’ve written two blogs, one about the cost of an exception and the second about using Dynatrace to evaluate exceptions and logs. Both of which discuss the great insight we can see quite easily with Dynatrace.

In the visual below you have a case of neglect. Yuck! We have broken windows everywhere. The visual is a snapshot of almost 1 million exceptions that our PVT test, which runs for about an hour. This test had about 4100 unique user sessions who performed about 56,000+ page requests. You saw what I wrote right? I said 1 million exceptions from an hour worth of user activity only by 4100 unique user attempts. That’s over 230+ exceptions a second. Holy smokes batman!

I call out the broken window theory because I see this problem of throwing exceptions as epidemic in the product. Fortunately, I’ve got the Flex team starting to begin looking into why are we throwing so many exceptions. We are wasting a ton of memory and CPU time by throwing these exceptions. Our goal is to get to no exceptions. This could, should and will become the standard going forward.

The visual below is also from the same PVT test. What we are showing here are all WARN, ERROR and SEVERE log messages (stack traces) that were raised during the test. It’s about 1100 log messages, which is about 1 log message every 4s. Not good…and yes another standard that we need to address.

There’s really no reason to create a “Chicken or Egg” kind of problem with this investigation. For simplicity, I’m going to say that I have a transaction in question. I’ve done an analysis of “Transaction X” looking at my baseline results and my regression test results. I see that “Transaction X” took a 50% hit. The transaction went from 3s to 6s. Not a big deal, but big enough for it to be on my top regression issues list.

First thing I want to do is map the data in Dynatrace with the data in Galileo. What’s interesting is that Galileo presents the Timestamp and Response Time. The Timestamp looks to be the end time. Dynatrace shows the Duration in the container (which should always be equal or less than Galileo) plus the start time. You have to do a little bit of math to figure whether the PurePath in question matches the individual sample. Why am I doing this? A couple of reasons. First, it gives me confidence that the data is correct in both systems. Second, it gives me both the CPID and the user in question so that I can use that information for profiling. I would always profile with the system at rest. Even that’s not exactly necessary as we can see “Suspension Duration” in the Dynatrace profile to see whether we were waiting on Garbage Collection. We can also look at the Method time to see if we are ever waiting to get a Database connection.

Below is my Galileo details view as well. I match the Start/End times plus the duration. Now I have the user’s information and the CPID.

Before I go into any kind of profiling exercise, I want to review as much of the data that I have available in the Dynatrace recording. I start with the PurePath of course. This gives me an idea of my hotspots of where to investigate. In this case I can see the class ViewQuestionsActions is where I’m spending my time. Interesting enough it’s a Struts call. So most likely I know that I don’t have enough information to make a call as to where my problem is or is not.

Next I dig into the tier breakdown so I can be confident of where to spend my time. If I see a lot of time is in Struts, but a ton of time also in JDBC, it could be that my Struts action is being masked by database latency. In the example below it looks like I’m spending about 5s in DB time, but close to 79s in Struts time. I wonder why.

I want to know where to look next. I obviously could go do a deeper YourKit profile with an emphasis on the ViewQuestionsAction. I want to be absolutely sure of the data. I can dig into the Methods view next for confirmation. Interesting enough I see 1 execution which accounted for 84 out of 89s of runtime. The key is understanding what this code is doing and why.

I still want to see where my database time is being spent.

Unfortunately I have an issue where I don’t see where the time is being executed. I do have the knowledge that we always reference Proxy9 when we prepare and Proxy10 when we execute. We can do a quick filter on the word Proxy and viola we see the statements in question. Note that we always prepare before execute. So it’s a guarantee that the Proxy10 that falls in question with no text is the statement from Proxy9.

So now I have the 4s+ of JDBC time narrowed down to this query referenced above. I have my class in question above. I have 2 optimization exercises to focus on. The 84s is obviously my priority. The 4s query is just as alarming.

What Should I Do Next

As I mentioned above, there really are 2 optimization exercises. We really need to get to the bottom of the latency exhibited in the ViewQuestionsAction. Obviously part of my problem is that I’m using an Extreme data set. So I have some kind of workload variation anti-pattern that gets worst and worst with the bigger the data set. If the problem was more uniform or static in nature, I wouldn’t see the transaction in question perform better with smaller data sets, which is exactly what we see.

I’m definitely going to run a YourKit profile with as much emphasis placed on that class and related classes. I could do the same with Dynatrace, but it’s not necessarily the intended purpose of Dynatrace. The Dynatrace tool helps us narrow down where to investigate forensically. Not only would I do a deeper profile with YourKit, I would also do a deep code review to better understand what are we doing functionally.

At this point I’m focused on decomposing which anti-pattern is introduced plus what functional problem we are trying to solve. If I truly understand what we are doing from a code perspective, I would then begin a refactoring exercise. If I didn’t understand both functionally and technically what we were doing, I would bring the problem to my architect or development colleagues for their input.

With regards to the second optimization exercise, I would start by investigation more about the query in question. We kind of luck out because it’s a single query executing that consumes 4.2s of time. So it’s not like I have to look at DAO anti-patterns. I basically can focus my attention on query tuning exercise.

JUnitPerf doesn’t complete the story. For one thing, JUnitPerf does not provide browser characteristics. Second, JUnitPerf only gives you part of the picture from an API execution. It doesn’t provide the layers of statistics that are inate to our MVC (Struts/Servlets). We really only see the lower level method execution. Third, in our own laboratory we don’t have the same maturity with JUnitPerf that we do per se with HTTP (LoadRunner) or even Selenium.

HTTP simulation isn’t always our best course of action. For similar reasons as JUnitPerf we do not see the browser characteristics as it’s one layer removed from the browser. Second, our agile development process doesn’t always leave us enough runway timewise to test over HTTP. Third, not all scenarios are run over HTTP. They could be command line oriented or service-oriented.

There is a key point about HTTP simulation that we can’t deny. That key point is it’s by far our greatest, most reliable and stable testing capability in our performance lab today.

JUnitPerf: When to Use It

Years ago a childhood friend of mine by the name of Andy Glover wrote this article about JUnitPerf. In the article he calls out that JUnitPerf is a valid testing tool, but it’s not necessarily the most accurate tool. I agree with Andy whole heartedly. I’ve always seen JUnitPerf as an Agile SPE tool used to faciliate feedback, but not as the quintessential tool for performance feedback. SPEs should leverage JUnitPerf during the development lifecycle for cases when a critical scenario cannot be simulated in its final shape or form. If a scenario is HTTP driven, or even command-line or services-oriented, but is not entirely ready for showtime, JUnitPerf should be leveraged as a stop gap for giving feedback.

“While JUnitPerf is a performance testing framework, the numbers you place around it should be considered rough estimates…”

Selenium: When To Use It

We might have become victims to our own performance goals. I’ve seen a lot of Selenium tests over the past 18 months. In many cases I’ve been promoting heavier use of Selenium to accomplish our testing goals. But why? Selenium is not the be-all…end-all of simulation tools. It has a purpose and we can’t lose sight of what that purpose is. Selenium executes the browser in the same intended manner a user would execute a browser. Therefore, you use Selenium as your test driver when you are predominately concerned with what is happening or executing inside the browser such as rich client-side interactions.

Justifying the use of Selenium because the scenario is executed via the front-end isn’t enough. We really have to breakdown the implementation from the perspective of which layer is executing code. If all that’s happening in the browser is rendering, then it doesn’t necessarily warrant a Selenium script. In a case like this, the code is predominately executing server side, whether that be in the container or in the database.

What happens when the browser is responsible for handling a voluminous amount of data to present from a DOM perspective? For example, maybe we aren’t executing code in the browser, but we return a 4mb data set to render. Isn’t that a problem? Absolutely and might require extra work to accomplish. It may make sense to simulate using two tools, with the emphasis being placed first on the server-side code and secondary emphasis placed on browser rendering.

HTTP Simulation: When to Use It

In our world, we use LoadRunner for HTTP simulation. So for the context of this section, it makes sense to use LoadRunner in the first place if the ultimate method for the scenario is over HTTP. Second, when the state of the code is ready to be executed over HTTP. Third, when there are attributes of the simulation that are best tested using our HTTP testing infrastructure such as timed tests, conditional abandonment, concurrency, workload throttling, etc…

We might find that HTTP simulation ends up being the home for 99 out of 100 tests we do. That’s ok in the long run when http simulation is capable of providing server-side feedback in a timely manner that does not jeopardize the browser experience. The key is timely feedback with our application. As an agile SPE shop, we have to be able to provide instant feedback to our teams so that the transition or even the next sprint can be planned appropriately.