My Soapbox on Automation

It’s been several years since I posted my blog about Selenium…More than Just a Mineral. As a team we have adopted Selenium to a degree, meaning we solved a number of scripting problems, use it for our experiments and even have a lightweight integration for kicking off Selenium tests with Fusion. There are a lot of problems we haven’t solve yet. We basically have to be committed this year to solve these problems for the betterment of our performance goals.

So I’m going to keep this blog less “soapbox like” and more goal oriented. The items that I’m going to cover in the sections below are my initial notes on MBOs that will be going in the direction of the team. They are things that I see as mission critical…essentially problems we have to solve this calendar year. By solving them, we will be a better performance engineering team and will have better data and testing capabilities to make our work more meaningful.

Selenium Top Pages

We need a starting point. The starting point has to be identifying our most important front-end pages. This could become an exhaustive list of use cases “eventually”, but right now we need to keep these use cases very simple. I suggest that in the end we go back to our performance engineering roots and turn these individual page requests into deep dive “performance scenarios”.

  • Login Page (pre-authentication)
  • My Institution Page (module view)
  • Course Home Page (module view)
  • Grade Center (Full Grid Page)
  • Discussion Board Pages
    • Forum Summary Page
    • Thread Summary Page
    • Thread Detail Page

The list might seem small. It might seem biased. In both cases it is. Why am I choosing these pages? They are critical to the user experience. They are historically under-performing pages. They are all well-adopted pages, meaning every single page that is analyzed has the potential to improve user experience or degrade user experience depending on the responsiveness of the page. Look at our testing models and you will see that all of these page requests (other than Grade Center) has a high distribution percentage. They have a high distribution percentage because of their adoption.

In the next section I’m going to cover my notes about how do we proceed with automating these page requests. For the most part we have Selenium coverage on these pages, or at least should have. The real question will be how much will we need to change the coverage? I’m sure we will get the obligatory next question, which is when can we include other pages that are just as important?

Expanding Additional Page Coverage

Let today be the starting line that propels you to think about your page coverage. As you can see from the list above, it’s heavily skewed toward Course Delivery. Second, it’s skewed toward one teacher page (Grade Center Full Grid) and one univeral page to both teachers and students (Discussion Board Thread Detail Page). Yes, we have all of these other pages which are just as important, but there’s still a ton missing. That’s OK…we don’t need a massive coverage model right now. What we need are some simple directions that we can execute on.

That doesn’t mean we can’t start identifying pages that are important to each of our teams. So here’s what you should do:

  • Start by identifying the individual pages that are most important to you and your team
  • Justify why these pages are “so important” with a blog
    • Your justification should be empirical
  • Put together a short plan on when and how you will move it into our coverage model

Flushing Out the Details of the Page Requests Chosen

A few weeks back I was under the opinion that PerfEng shouldn’t be in the business of writing Selenium scripts. I’m not entirely convinced that we shouldn’t be in that business. I broadcasted in front of the entire team that QA Automation should write all of the scripts and we should leverage them. There were a few naysayers in the audience that said it would be risky for us to attempt to leverage their scripts. I’ve had a couple of weeks to think about it. Here’s where I’m at…

I don’t way say we have to choose right now. I think we need to start by drafting up requirements for our Selenium page requests. This involves drafting requirements about the page requests and the data interactions. QA automation uses it’s sample data. We use Datagen. Maybe we both are wrong. Before we make a decision, we need to have some discussions with QA about their scripts. We need to figure out whether we can use the scripts as is. We need to determine whether we could put together requirements for automating and have QA Automation fulfill them. We need to determine whether we could do the automation ourselves and maintain the automation.

We need to be creative and not lock ourselves in a box. Maybe we need to create course packages that are 100% dedicated to our scripts? These course packages would include a more robust data set that leverages the course customization and content design capabilities in a more realistic fashion. We all know that datagen today is about data creation, not data presentation. Maybe we simply need to buckle down and design content that’s oriented specific for our Selenium scenarios? All I’m saying is we need to keep an open mind and consider alternatives.

Integration with our Test Infrastructure

I was literally about to call this section “Integration with LoadRunner”, then I realized boy would that be short-sighted of me. A lot goes into standing up an automation technology, especially in our lab. We need to do this in such a manner that we solve problems, rather than creating problems. The problems we need to solve are not simple, so it’s in my best interest to call-out that I know it’s going to take some time and resources to solve this problem. I get it and promise to be patient.

Next Generation Selenium

It’s my understanding that Selenium 2 is finally on the market, which is vastly improved for FireFox and IE, but now includes support for Chrome. Our own QA team has been working several problems and can serve as a big resource to us if needed.

We need to start with a project that evaluates Selenium 2. This project has to be short in nature (less than 1 week). It needs to be able to determine what changes we need to make to our scripting to migrate to Selenium 2. It needs to solve browser support for IE, FF and Chrome. We can’t take the easy way out and only support 1 browser. It has to be all 3. We have to have our logging integrated and be able to capture true performance timings.

Integration with Fusion

Our integration has to be tackled to solve the many problems of automating Selenium. We need to be able to dynamically test from any system in the lab. Taking the approach of setting up Selenium systems like LoadRunner Controllers doesn’t make sense to me. Why? Because it’s a software code base that changes regularly. We need to be able to patch when necessary. I see this as the first problem.

Second, we need to be able to launch tests in a more effective manner via Galileo/Fusion. We need to be able to see what coverage we have or don’t have. I’m not sure how we solve this problem, but what I am really asking for is way to see the tests we have from a drop-down menu by release. I think the drop-down is critical because we often forget or fail to look for our coverage. When we get new team members they have no idea where to go. Having the drop-down is clutch because it makes it simple to select a “known test”. I’m up for debates on this request…

Third, we need to solve the processing of test results and integration into Galileo as a test result. This has been our thorn for multiple years. We haven’t solved the problem of processing the results, storing in Galileo and even presenting visuals of the data. We have to solve this problem as part of this work. We can’t accept looking at a log file as a valid solution.

Integration with LoadRunner

One of the problems we have to solve from a Load Test perspective is being able to accurately measure end-to-end response times including browser performance as part of our load tests. It’s not something that is often done as part of performance testing and benchmarking. I think it’s the missing link for a valid performance test.

Here’s what I’m envisioning…when defining a load test, we pick and choose what Selenium scripts we want to run. We choose the sample frequency inputs (when, how many, priority, etc…). We can also choose dimensions, or any other test attributes. We use the Selenium results as a differentiator of data. First, it gives us a way to study true performance characteristics under load. Second, it gives us the ability to look at response time characteristics under varying load levels. Probably one of the most important points is that we know our LR scripts are slimmed down. They don’t do full HTTP loads. Selenium requests would force the full HTTP payload into the browser.

Dynatrace Integration

We have to be able to measure browser performance. It’s crucial to our performance framework. The goals of our framework include generating load, but also measuring load. Our browser plugins for Dynatrace give us the ability to measure deep into the browser. We need to be able to provision this in an automated fashion. We have solved this problem via LoadRunner. Now we need to solve this problem with Selenium.

This involves both the manipulation of our script infrastructure to turn Dynatrace on/off at any time. It also means being able to initiate a measurement session, as well as packaging the results into Galileo.

Build to Build Regression Suite

We might have been a little ahead of ourselves a few years back with the idea of an APDEX test. Or the pessimist would say we missed the boat entirely with APDEX via LoadRunner and should have looked at browser based technology. The whole goal of APDEX was to analyze performance in conditions not under load, as well as under load for the purpose of comparing unequal page requests. That sounds a little confusing, but essentially we wanted to be able to create a score card which any transaction could be compared to another even if they were completely unrelated.

APDEX never really took off. It created a lot of work, a lot of data and a lot of a mess. Now’s our chance for take 2. I’m not saying take 2 of APDEX scores, but rather take 2 of running a build to build regression suite of key transactions in a system not under load, as well as a system under load. I really look to Chris to incubate the idea further. I’m looking for something as simple as a Selenium test suite that runs build to build in a continuous fashion. We get the ability to use the data for comparative purposes (SUT vs. Non-SUT). We use to data to create control charts (b2b regression). We use the data to see dimensional comparisons. We use the data as an indicator of a degradation trend (same as control charts).


One thought on “My Soapbox on Automation

  1. Pingback: Coming Back to Browser Performance Testing « Seven Seconds

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s