Velocity Workshop Complete Web Monitoring

Information About the Session

Metrics 101: What to Measure on Your Website

Sean Power (Watching Websites)
9:00am Tuesday, 06/22/2010
Web Performance Ballroom AB

Please note:this is a workshop, for which there is an additional fee required to attend.
This session will help you build a complete web monitoring strategy. We’ll cover the many different metrics you can collect, from latency and uptime to usability and navigation – and show you how to tie them to the goals of your web business.

Taught by the co-authors of O’Reilly’s Complete Web Monitoring, this packed workshop will look at every part of your online measurement strategy, with a particular focus on what web operators need to know. We’ll cover:

  • The elements of web latency
  • Strategies for collecting end user experience
  • How performance and availability fit into the rest of the monitoring picture
  • Linking performance to business metrics like time-on-site, conversion rates, and engagement
  • Measuring sites you rely on, but don’t control
  • How to roll up measurement data to share it with other stakeholders

About the Speaker

Watching Websites
Sean Power is a consultant, analyst, author and speaker. He is the co-founder of Watching Websites, a boutique consulting firm focusing on early stage startups, products and non-profits as they emerge and mature in their niches. He has built professional services organizations and traveled across North America delivering engagements to Fortune 1000 companies. He helps executives understand their competitive landscape and the future of their industry. He has done technical edition for Troubleshooting Linux Firewall for Addison-Wesley and co-authored Complete Web Monitoring for O’Reilly media with Alistair Croll.

Sean has had first-hand experience creating and implemented social computing strategies with larger companies like MTV and smaller startups like Akoha. He is active in the social computing space, using Twitter and blogs as his communication platforms of choice. He often speaks on the subject of product acceleration, measurement or social computing in clinics, workshops, presentations and one-on-one training.

Notes on the Session

The session started off with Steve Souders kicking off. He mentioned that the conference finally sold-out. It’s quite amazing how many people are here for the workshop. It feels as though there are 2X more attendees this year then in the past. Originally, Alistair Croll, formerly of Coradiant was going to give the presentation, but he bailed. So his co-author gave the session instead. The goal of this session is to really understand the basics of web monitoring. Apparently Sean is a former Coradiant guy as well. He ran the services division at Coradiant for a number of years.

Note to Self: It might be a good idea to (a) pick-up their book and (b) would we be willing to bring them in as consultants.

New book called Web Operations from Jesse Robbins coming out today. Might want to pick this up well.

Power’s started the session off discussing the cost of Downtime. His example was that every hour Amazon goes down, it loses 1 million dollars. Typical consumer sites lose about $50k. He had an interesting slide about availability percentages and the total cost or loss of revenue.

Next he went into role of planning. Study data quarter over quarter…year over year. Trends are used for strategic planning. Automation is absolutely critical. Not just for operations, but for monitoring. Then we need measurement for optimization.

  • Everything starts with a baseline
    • Account for what is happening (Accounting Analytics)
      • Know what is worst
      • What you can optimize
    • Make it better through optimization
      • It’s a cyclical process: collection, reporting, institutionalizing, KPi/ROI, Repeat…Repeat
  • Have to do more than just collect
    • Must tie to something or some other aspect of business
  • Understand your business goals
    • Amazon’s goal is to maximize sales
    • Four types of businesses on the web
      • Transaction Sites
      • Collaboration Sites
      • Software as a Service (SAAS)
      • Media Sites

Slow sites suck! Obvious…but let’s get to why. Sites that are slow affect conversion rates. They are less likely to keep their user’s loyal. Poor performance can costs money (refunds or service credits). Customers may find other ways to reach you.

How TCP can affect performance. Used an abstract example of the Postal Service. Follow the rules…put a stamp, get to mail box before 5pm and put the right address. It just works…TCP is no different then the mail. There are rules and a natural order/sequence to everything. TCP creates an E2E link. HTTP has rules as well for requesting web objects.

Random Note: Was searching around SlideShare for this presentation, but found this one instead from a few years back.

Where to Measure

  • Three tiers of data
    • WAN Accessibility: Can users reach the site and how long does it take to process?
    • App Functionality: places and tasks
    • Tiered tests
  • Use analytics to drive synthetic tests.

Synthetic Testing

  • Use internal systems like Naggios or HP Openview
  • Use a monitoring service outside of firewall
    • Should we consider using the SOASTA appliance from the DR location to run both web/HTTP tests and functional/Selenium tests from outside of the network?
      • Browser puppetry
  • Synthetic monitoring is simply not enough

Math Rules

  • Averages Suck…little play on Millsap’s skew (Don’t use averages)
    • Let’s actually stop using averages on the team.
  • Use percentiles and histograms
  • Use traffic requests per second

Another Cool Idea
Show the histogram of a “business process” or clickpath. How cool would that be in which we show the histogram for CPIds?
Consider using an image of the Count from Sesame Street for presentation at BbWorld.

You know how we show histograms of individual transactions? Would it be possible to take full CPIDs (add all transactions for a user in a CPID) and develop a histogram for those? Let’s say that CPID 1001 (taking an assessment) is performed 350 times. It would be good to show a histogram of the entire CPID (total time).

Another view would be to show scatter plot of the CPID over time as well. Represent the dot by the total time and the position of the last time. If we didn’t do a scatter plot, we could offer a line chart from start to end.

What do you think?

Target Metrics for Audience
Speaker suggested APDEX…not really interested in going down that route

Tools Mentioned

  • Site Meter: Like Google Analytics
  • Net Optics: Network Tap for User Performance Monitoring
  • Tea Leaf: Customer Experience Monitoring
  • Dotcom Monitor
  • Aternity
  • Beatbox
  • Atomic Labs
  • Moniforce (Oracle)
  • Unica
  • IBM Coremetrics
  • Web Page Test
  • Jiffy
  • Juice Analytics
  • Dashboard Spy

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s