Monthly Archives: September 2008

Outliers Should Never Be Ignored

Have you ever wondered why outlying transactions are often displaced or ignored? For a long-time I never gave it a second thought. Everything I had read up to a few weeks ago told me to avoid focusing on outliers as worthy transactions. Even my former mentors and respected peers told me to eliminate outliers. Take a look at this article from Scott Barber and you will see he too discourages outliers. He uses his article to talk about finding outliers, which he defines below, but talks about removing them from the result set.

“Outliers are atypical (by definition), infrequent observations; data points which do not appear to follow the characteristic distribution of the rest of the data. These may reflect genuine properties of the underlying phenomenon (variable), or be due to measurement errors or other anomalies which should not be modeled.”

I used to ignore outliers during my performance tests. Now I consider them the most important pieces of data that I can find. They are the first thing I look for when reviewing the results of a performance test. Why might you ask? Well in my opinion outliers tell you how bad responsiveness of the application can really get.

Let’s put it in perspective. Assume I am measuring login performance to the application. I have 100 independent samples of the login transaction. I review the data and I determine that the Mean Response Time is ~1.5s. I also find out that the Mode Response Time is 2.2s. These are all fine transaction response times. What if the maximum response time was 600s? That’s 10 minutes right? I can’t imagine someone willing to wait 10 minutes, but then again I found myself going to the Deli counter at Safeway this weekend when I swore I would never go again. So anything is possible.

600 seconds is a ridiculous outlier. It possibly could have been worse. There might have been a few other samples that performed nearly as bad. If we ignored those data points because our average happened to be 1.5s and our mode was 2.2s, we are more probable to miss a performance issue.

I think it’s important to remember that a performance issue doesn’t need to be a trend in order for it to be a performance issue. What I mean is that performance is relative to the person experiencing it. So if even 1 user experiences a cruel amount of latency then a performance problem exists.

So knowing this, I think it’s in our best interest to figure out the causes of latency inflicted on an outlier transaction. Let me leave you with this abstract example.

This past Saturday my family moved from our house in to a new house. I’ve been dealing with ordering cable television since Friday of last week. It’s been an incredibly frustrating process. When I called on Friday I waited 25 minutes for someone to speak with me directly. My goal on Friday was to cancel my cable at the house I’m selling. On Monday I called the cable company again. This time my goal was to add service. I waited less then 2 minutes to speak with an operator. I called at exactly the same time which was 6:15pm on my way home from work. I was really surprised at how fast my call was answered. So I tried a couple of experiments on Monday and Tuesday. The experiment was to see whether the cable company purposely made me wait when I wanted to cancel my service and provided quick service when I wanted to add or upgrade service.

So here’s what I did. I called the cable company using 2 different phone lines when I got home. On one phone I chose to cancel service. On the other phone I chose the option to add/upgrade service. I did this 2x on Monday and 1x on Tuesday. Here’s what I saw. In the three times I chose to cancel service I waited 25 minutes, 17 minutes and 11 minutes. In the three times I chose to add/upgrade services, I waited 2 minutes, 4 minutes and 1 minute.

I was concerned that maybe the cable company had different people serving in different roles. Maybe they had more people handling new/upgrade services versus canceling service. So the first question I asked was whether the operator answering my call could do either function. In all 3 cases, the operator said they could do both and that there was nothing in place at a system level in which an operator took one type of request versus another. It was entirely based on the availability of the operator. I was equally concerned that maybe the time of day I called was an issue. Two things about that. First, I called at the same time using different phones, so this would be somewhat contradicting. Second, I called at different times concerned that call volume was an issue. The times were nearly identical.

So what do we make of this story? Is it fair to say the cable company was creating an artificial performance issue on purpose? It could be, though it’s very difficult to prove. Does my story have nothing to do with outliers at all? Well actually it has everything to do with outliers. I was really upset waiting so long. I was so upset that I created my own experiment to see whether there was an actual conspiracy theory involved with canceling cable versus adding cable service. I’m starting to believe that the conspiracy theory is more likely true then not. The big issue with my cancellation experience was that I was incredibly frustrated that I will forever equate response time and throughput issues with the cable company…and when you think of cable companies, you want to think about fast, reliable service. Just like the deli counter example from a few weeks back, an outlier can be real problematic. The outlier can become an antagonist causing some pretty severe issues. These could be PR related or something else…

That being said, if I’m an outlier doesn’t my experience matter then?

The Bug Man…An Analogy of Sorts

A few months back I had an issue with carpenter bees. At first I though the issue was termites. Termites had been a problem at my house before. On the eve of our one year anniversary at what is now our old house (fyi…my family and I just moved a little over a week ago) we discovered termites in our laundry room. The worst of it was they hatched. The only good thing was that they were isolated to a small area of the house. There was little damage, but would necessitate years and years of termite maintenance plans.

So when I saw a bunch of wood dust on my deck, I immediately started to fear that the termites were back. Not only were they back, but they seemed to be immune to our termite control system that we had spends thousands of dollars on. The first thing I did was call up our termite control company. I’ll keep them nameless for now as I don’t want to send any bad press their way. They came over within a day and diagnosed the problem as carpenter bees and not termites. They also told me that my plan didn’t cover carpenter bees and worst yet that they didn’t cover carpenter bees under the pest plan. They did tell me one thing…guess what that one thing was? They told me that our carpenter bee problem was really bad.

At this point I’m kind of up a creek without a paddle. I have an insect problem that needs to be eradicated. I have no experience with eradicating the insect. I have no formal idea about how much damage my deck has sustained. I have no idea how bad it could get and how soon. Worst of all, at the time I was trying to get my house ready to put it on the market to sell.

I did some research and learned everything there was to know about carpenter bees, well that is as much research as I could find off of two or three queries from Google. I learned that they burrowed into the wood. I learned to look for round holes in wood that turned into 90 degree angles once inside of the wood. I learned about spraying certain pesticides and puttying up the wood so the bees were trapped and died. It turns out that I had an entire colony of carpenter bees. While doing all of this treatment I noticed a bunch of bees were hovering around a dying tree in my neighbor’s yard. The bees were two-timing both of us. They were nesting in the dying tree and eating on my deck. You know they old expression that you never sleep where you eat. Well, that’s apparent with carpenter bees as well. Eventually we got the county to remove the tree, my deck was fully treated and patched by me and the problem went away.

There’s something more though about the story. Remember how I said my pest control company that I had a termite plan didn’t have coverage for carpenter bees? Well I forgot to mention that I had a general pest control contract with them as well. The pest plan had hundreds and hundreds of families of insects and bugs that they covered on their plan. There were three in general that was prominent in my neighborhood that they claimed they couldn’t treat: mosquitoes, carpenter bees and wasps/hornets. Looking back, I paid about $65 every 2 months for pest control. The main pests that were troubling my house weren’t under my plan. I don’t think any of the hundreds of bugs on their list were indigenous to my area. Certainly, mosquitoes, carpenter bees and wasps/hornets were indigenous to my backyard. So at that end of day, I’ve got to say to myself what exactly am I paying for? I still have wood damage and tons of mosquito bites, not to mention the hornets are so big that my wife is scared to let my daughter play in our backyard.

In light of this little story, I think there’s something to think about. That’s specifically about understanding your audience. I don’t think our pest control company understands it’s local customer. First off, it’s one of those giant franchised company that exists all throughout the United States and Canada. Sadly they take a one size fits all approach to pest control. What works in New Mexico must work in Maine. Understanding your customer or your intended audience is the best way to succeed. It’s what keeps them coming back and wanting more. It makes them want to refer you to their friends, neighbors, etc…

I’ve since moved and I have a bug problem. I’ve got mosquitoes and bees (fortunately not carpenter bees). I even have a few chipmunks and mice prancing around my yard. I’m going to approach these problems a little differently. I’m going to work through these issues like I did the carpenter bees myself and try to find a company that can specialize in my needs. Let’s see how it works out…