For a while now we have been talking about better browser performance testing. We have gone through multiple years of analyzing measurement tools, scripting frameworks and browser performance methodologies. After all of these years of effort and evaluation, we haven’t moved the needle in terms of developing a purposeful browser based performance test methodology.
I will note that we have made some initial progress from an HTTP perspective. What am I talking about? Well one thing that Chris has done since he arrived is started the process of full page loads. I had called out the need for full page loads back in this blog entry a few months ago. Chris and Raja have made some good progress by handling the Course Home Page. I’m not sure if he has a project plan on some of our other key scripts or not. Hopefully, he does…
As I look back at that blog entry, I realize that I was still latching onto Selenium. In fact, I called-out that Selenium 2 was just released and that the QA team was doing an evaluation.
Let’s step back for a second and ask some serious questions…
- What are we trying to accomplish with browser testing?
- Do we really have to use the same tools as QA?
- Do we have a set of use cases that would be initial targets for browser performance testing?
I have never really called-out what’s important to us. There are so many things that are important, but we should be able to narrow the list down to a subset of key data points:
- Time to First Byte to Document Complete
- Page Size
- Object Size
- Object Count
- Cached or not Cached
- Specifically, what’s changed during a “Repeat View”
Any tool that’s able to simulate our request, while at the same time capture this data, plus the HAR (HTTP Archive) which we can use in tools like dynaTrace is ideal for our goals. We want to better understand the first question better than any tool. What’s the best way to do that? I would argue that experiencing it first hand is the right way. Manual testing doesn’t scale. It’s not scientific. It’s not conducive to our team considering we are spread all over the world. We need a way to automate, yet experience. Do we buy really expensive, high-resolution video that captures a computer screen as a Selenium test executes? Doesn’t seem likely we would a) get the money or b) have the infrastructure to make it work.
Then there’s the argument about Selenium as a performance tool. It can execute a browser, but it doesn’t really have the infrastructure to give us the data we want. We would have to invest more development time from our team. Plus, we have the constant debate about duplicating work. Truth be told, I don’t believe Selenium gives us time from First Byte to Document Complete, but I could be wrong. I usually am…
OK…so we need something that can be scripted and automated. We need something that can allow us to “feel” the user experience. We need something to provide us better data and more accurate data. We need something that would better integrate w/ our tools like dynaTrace. We need something that won’t be 100% redundant, but could be similar from Automation. We need something that can be centralized so that geography is not an environmental factor.
I would argue NOPE! They don’t use the same tools we use is the simple argument. That’s the immature way of looking at things. I would say we don’t want to be 100% redundant, but we want similarities. The way we use Selenium for instance doesn’t match the way they use Selenium. We already know that they are not interested in taking our requirements to make adjustments to their Selenium framework. They also are against us making changes directly. So there we have it…We won’t be using their automation and they won’t be developing our automation. With that out of the way, it’s safe to say we are not bound by working this problem together.
So with that said, we need to work the problem to fit our needs first. We need to solve the problems I listed about…
|OK…so we need something that can be scripted and automated. We need something that can allow us to “feel” the user experience. We need something to provide us better data and more accurate data. We need something that would better integrate w/ our tools like dynaTrace. We need something that won’t be 100% redundant, but could be similar from Automation. We need something that can be centralized so that geography is not an environmental factor.|
Thinking about it even more, how great would it be if we could present a tool or tool artifacts that could be included in a ticket for engineering, or visual feedback to the designer or UX specialist? To quote Chris Farley…
|Chris Farley: Remember that time you were with the Beatles?
Paul McCartney: Yes.
Chris Farley: That was awesome.——————————————————————————–
Chris Farley: Remember when you were with the Beatles and you were supposed to be dead, and there were all these clues, like you play some song backwards and it’d say, like “Paul is Dead” and everybody thought you were dead and, um, that was a hoax right?
Paul McCartney: Yeah, I wasn’t really dead.——————————————————————————–
Chris Farley: I think we got time for one more question. Remember when you were in the Beatles and you did that album Abbey Road and at the very end of the song, it went: ‘And in the end, the love you take is equal to the love you make’. You remember that?
Paul McCartney: Yes.
Chris Farley: Um, is that true?
I think I’ve said it over and over again that we use Selenium when we are attempting to see true user experience time and browser behavior. So our use cases should be such that we are focusing on the ills and differences of browsers and their effect on the user experience. I have a few thoughts as a starting point:
- Any page that loads modules (especially asynchronous modules)
- Any page that shows a page processing interface (aka…the GradeCenter)
- Pages that demonstrate comparison views of CSS behavior
- Pages that have a potentially massive DOM
So, remember that time I said we should do more Selenium work? Yeah, I was wrong…I think we need something better like WebPageTest. Yeah…that would be awesome.