Tag Archives: spe

Relevance of Time on Task to SPE

Time on task is a simple way of studying the time it takes to perform a given function or task. It’s an effective way of measuring the efficiency of a workflow or design. Time on task can be quantified by studying the elapsed time from the beginning of a task until the end of a task.

More often then not the phrase is represented as a measurement within usability testing and human factors. There’s a fairly basic notion that if a user can perform X operation in Y amount of time, where Y is desirable, then the experience is better. That pretty much blends well with most of my beliefs and studies about page responsiveness. The faster a page responds, the more engaged and comfortable the user becomes with interacting with the page. Could you imagine a user actually complaining about how fast a page responded?

I have to admit, I have yet to work with a user who complained a page request was too fast. There have been some cases where users questioned the experience entirely. Speed was an attribute of their questioning. The context centered around the validity of data. In the few isolated cases that I can recall users questioning responsiveness, it was that the data returned was questioned. The page request may have come back in sub-second, but the data was disjointed or inaccurate. In this case the experience was muffed.

It’s not necessarily fair either to say that time on task is better when it’s faster. From an academic sense, there’s widely accepted research that suggests faster isn’t better. I’m by no means argue whether this is right or not. What I can say is that it makes sense to me that in many educational scenarios, it makes sense to determine the appropriate boundaries of time to complete a task beyond faster is better.

So Why Exactly is Time on Task Important?

Inside the software world, it’s important for certain tasks to take minimal amounts of time. Tasks which require minimal thinking, unlikely the exercise the brain are optimal candidates for short time on task measurements. Tasks that are performed redundantly by users are additional candidates, though with redundancy of task comes the opportunity for optimization of workflow. Critical tasks that can make or break adoption of a feature set are also candidates for short time on task. I believe they are important purely from the perspective that if a task is too cluegy to perform or simply takes too long, users performing that task are going to quickly become frustrated. The saviest of users will look for short cuts. When they can’t find the short cut, the either lose interest or abandon. Both cases directly affect adoption of a new task.

How Time of Task is Applied to Usability

Usability engineers use time on task as a core metric for observing the efficiency of a task. Elapsed time is often measured directly (stop watch or embedded timers) or indirectly via recording tools. Multiple samples of the same task are studied and analyzed. Often the data is presented in the same fashion we present data in Performance Engineering. Mean values (specifically the Geometric Mean) to complete the task, as well confidence intervals (UCI and LCI]) are studied to present a stastical view of time on task.

Initial Thoughts on Time on Task Relevance to SPE

I think the key piece of information that needs to be applied to SPE has to do with task efficiency. When it comes to responsiveness, we try to place a cognitive value on a task. The way we do that is apply a utility value for performing the task. We combine the utility value with a patience rating of a user. The combined utility + patience dictates the abandonment factor.

I believe time on task is in fact the missing piece of data that would make our abandonment decisions more meaningful. Come to think of it, really our abandonment decisions are guesses on what we believe will be the rational behavior of a user who becomes frustrated. It uses arbitrary response time factors to determine whether a user will become frustrated or not.

How exactly can we be the authoritarian on likelihood of abandonment if we do not have much context on expected time on task?

Steve On SPE Voting of Clickpath Probability Models

I’m by no means attempting to be Joel Spolsky based on the title of this blog. I just wanted a simple way to say I’m really interested in talking about clickpath probability models. I think we need to consider a more effective manner of gathering input from our sprint teams with regards to our probability models that PE delivers.

Our main challenge is finding the time from others to be more interactive with our SPE efforts. Everyone is busy, so finding the time to get all of these different constitutents to provide input is darn near impossible. So maybe we have to figure out a better way to use their time more effectively for the inputs that we really need.

From an SPE perspective, the two pieces of data that we really need input from our team constituents is with regards to the lifecycle of our data model and our probability models. I think the data model discussion is a little too complex for me to start writing about here. So for simplicity purposes, I will focus on our probability models. Below is an example of a probability model taken from 9.1 about Assessment creation using question discovery.

Assessment authoring is by far one of the more critical content authoring activities in the system. It is also one of the most expensive from a resource and storage perspective. The process of authoring a question is treated with a lot TLC by content creators. How exactly do I know this? Well, I don’t have exact empirical data that says 84% of all assessment creation activity requires intense critical thinking and lasts as long as 3 hours end-to-end. Whoa…just throwing out an example. I do understand that assessments are instruments for measuring academic performance. Given that our culture is to put more utility in the measurement process, it seems likely that this would be an important activity.

I’m a victim of the problem I’ve described above, as well as a contributor. I could have easily solicited input from our team consitituents looking for their guidance about the critical nature of assessments. Some of those folks would have given an answer like mine, meaning they would believe this is a critical process given the nature of measurement in academic worlds. Some of them and hopefully the right group of people would give me the answers that they have heard first hand by the users of the system, the ones who really matter.

Getting that information is really hard. Asking the question isn’t necessarily tough. Getting the answer is tough, especially when the interviewee doesn’t have time to answer you. So I want to propose we open up our issue a little further by the team. We need to figure out how to get more precision in our probability model. Precision comes from input from our team consituents who are closer to the real data. The real data we want will shape our probability models.

As SPE engineers we should be able to read requirements and then specifications to build out all of the states, decision points and actions in order to identify the possible model paths. We should also be able to infer from the artifacts our teams give us such as the MRD and specifications our own pass at the probability models that shape the weighting of our clickpaths. What we really need is a way to receive input about these models in an easy to interpret and participate manner.

I would love for our team to rack their brain on possible solutions. At the end of our SPE day, we have to feel confident that we:

a) Receive the correct input and feedback about how users will engage with the application
b) Improve our transparency so that teams value our contributions and efforts toward the development of the product
c) Encourage our team members to consider probability models as an important component of software development

Steve’s Thoughts on Making This Happen

There’s no way that I am going to write a 2-page blog without providing my own input on a way we solicit other’s inputs. As I stated above, I think one of the challenges facing Product Managers, Designers and QA Analysts is a lack of time. For us to expect them to review our documents asynchronously is not too wise. For us to expect them to have time to meet face to face or over WebEx isn’t all that likely either. So we have to find a way to give them an easy way to review our work in a meaningful workflow that demands little time and not too much thought (as a means of reducing time and effort). They need a way to approve or question our probability models. Most importantly, we should consider building a tool set that in due time they would want to adopt themselves and ultimately seek value from (Refer to The Dip by Seth Godin).

The tool I think of is inpsired by a blog that Nakisa herself wrote back in March when she talked about a tool that’s out there on the web called Web Sequence Diagrams. This is an easy to use visualization tool that models UML interactively from simple Wiki markup and subsequently creates UML notations.

Another tool that has inspired me is Cuzillion from Steve Souders which is another simple modeling tool, but this time for modeling the ordering and sequence of web parts of an HTML page.

Both tools provide a very simple interface to construct models for their purpose. I envision that we need a tool that gives us the ability to model states, decision points and actions. We then need a way to highlight the critical paths. In essence, each path needs a weighting system to differentiate probability. We need the modeling aspect and the visualization aspect, but we also need a way to solicit input. One way is through a publish and vote process where constituents could be sent a workflow to review the probability model and then vote on its accuracy. We could also give the ability to fix the model and then compare the two or more models head to head.

If I didn’t mention it before, if we built a tool like this, it would go into Galileo (our home grown system) and be used for other data oriented projects.

As I said before and will say again, we really need input from the team about how to solve the problem of getting input. The manner in which we get the data is up for debate. Let’s start debating…

Seven Habits of Highly Effective Performance Engineering

For those of you who know me, you probably wonder why I read so many books that have nothing to do with Performance Engineering and mostly have to do with management, organizational behavior and sociology. At any given time, I am typically reading 3 or 4 books in which the majority are always a business book and then of course I read at least one technical book. There is one book that I read quite some time ago from Steven Covey called the Seven Habits of Highly Effective People that I enjoyed most of all. While driving this evening, some folks on NPR brought up the book and it got me thinking. There are some characteristics that I believe define a good performance engineer from an average performance engineer. If you know me well, you know that I love what I do. It’s not just a job for me, but a career.

Learn SPE…Don’t Just Memorize It

As a young performance engineer I lacked any formal guidance. The team I worked on had little process or approach. It wasn’t a bad team. In fact at our old organization, it was one of the most respected teams in the company…almost to the point that my old manager was god-like in the eyes of so many employees at the company. Before I left, I joined a professional group called CMG and quickly learned about an interesting methodology called SPE (Software Performance Engineering). Below is a definition of SPE from Williams and Smith:

Software Performance Engineering (SPE) is a systematic, quantitative approach to the cost-effective development of software systems to meet performance requirements. SPE is a software-oriented approach that focuses on architecture, design, and implementation choices.

SPE gives you the information you need to build software that meets performance requirements on time and within budget.

The SPE process begins early in the software life cycle and uses quantitative methods to identify satisfactory designs and to eliminate those that are likely to have unacceptable performance before developers invest significant time in implementation. SPE continues through the detailed design, coding and testing phases to predict and manage the performance of the evolving software as well as monitor and report actual performance versus specifications and predictions. SPE methods encompass: performance data collection; quantitative analysis techniques; prediction strategies; management of uncertainties; data presentation and tracking; model verification and validation; critical success factors; and performance design principles, patterns and antipatterns.

When I came to Blackboard in the fall of 2003, the first thing I set-out to do was build an SPE practice. I’ve been moderately successful, which got me thinking as recently as 1 month ago that maybe the methodology is just too cumbersome for the team. I’ve been watching the team quite closely and I realized, I don’t think that the methodology is hard to understand. It’s quite practical and can be easily streamlined. Where I think the problem resides is that learning the methodology for most of the team has been viewed as a heavy academic exercise. Heck, I hand every new employee on the team a copy of Performance Solutions: A Practical Guide to Creating Responsive and Scalable Software and I think the book is so big and heavy (it’s a text book used by a lot of graduate school programs) that the content get’s lost. You tell me when the last time you snuggled up in bed to read a text book? Most likely it was college and I bet within 10 minutes you used to fall asleep while reading a book like this. The thing about this book is that the workflow and process for executing SPE is so simple and so straightforward that getting through the book shouldn’t be such a struggle. I’ve read the book 30 times in 5+ years. That’s 6 times a year or once every 2 months…

Research Daily

Another tidbit about me…did you know that I am a list guy? That’s right…every day I write a list of everything that I am to work on. I start the list in the car at stop light, at my desk and sometimes even at meetings. I work through the list each and every day. I don’t necessarily finish my list each day, but I get real close. Before having lists my days were quite chaotic and I wasn’t quite sure what I accomplished. One of the items that makes my list each day is a line item to read through Google Reader. I use Google Reader as my RSS reader. As of today, I have 186 subscriptions and the list is growing each day. I recommend that you download my OPML file (google-reader-subscriptions_110308) and use it as a starting point. Every day, usually around 12:30pm I scan through the blogs from the previous day. Some blogs are bogus and I skip over them…others are pretty cool and I tag them. Lately I’ve been starring them in Google Reader, posting them to Scholar and sending it out to the team. Every day we need to learn. Blogs are one way. Books are another…remember that the library is free. So when you think that you are having a bad day stuck in meetings or whatever…take 30 minutes to do some reading about our discipline of performance engineering.

Share Your Experiences

Each and every day we run into problems or learn something new (ie: habit #2). We need to be more vocal about our experiences so that everyone on the team can learn. Our blogs are meant to be our space for sharing these experiences. I’m really starting to believe that if we don’t have an entry a day, it’s a wasted day. That might sound odd, but I really mean it. Let me throw in a caveat. Some days you simply don’t have time to write. That happens to me all of the time. In those case, I save my writings for the days that I can contribute a posting. Often, I rail off 3 or 4 blog posts in a day as a way of catching up for those days that I’m too busy to post. What does it mean to share your experiences? It really depends. I tend to talk as though my reading audience is really a listening audience. My blogs are somewhat conversational and impersonal. I like to put lots of links in so that if my point(s) aren’t as obvious as they should be, the reader can navigate to one of those links for additional context. I also like to put pictures in my postings as often as I can. Sadly, some of the visuals I want to post are too time consuming to put together, so you might get a low budget visual. As of most recent, I’ve been doing video blogs so that I can review a topic or tool set. This is something that I want to make more a part of my weekly routine. Watching a video can be extremely powerful…especially with embedded audio.

Show Your Work

This is really an extension of #3 Share Your Experience. For this point, I want to share a quick story. In high school, I had a Math teacher named Captain McIsaac. My high school was originally a feeder school for the Naval Academy, Coast Guard Academy and the Merchant Marine. So we had a lot of older teachers who used to be former Navy. Well anyways…Old Cap McIsaac was an interesting guy. He looked like Ted Kennedy’s twin and probably scored the same on most of his breathalizer tests. He was a terrible Math teacher. Most of us thought he was awesome because he would give us the answers to the questions on our exams during an exam. We never had to show our work. That’s great for kids who cheat off each other. I have to admit…looking back the guy was terrible. He didn’t hold us accountable for our work. It showed in all of my Math classes after Cap’s class. I did well because I love Math, but it takes an awfully long time to break bad habits. You can pick-up a bad habit in seconds, but it takes weeks…sometimes years to break a bad habit.

There’s an important reason for showing your work…actually there are multiple. The number one reason is so that you personally can spend the time reviewing what you did and explaining it to your peers in a visual manner. Don’t worry if you change your ideas…you just write new blogs. The second reason is that we are a global team. Everyone on the team should get the opportunity to learn from other members of the team. It’s a good way to get feedback and share work. The third reason, which is sadly a bit lame is that our days become so busy, that sometimes we need to be able to comment on a blog rather then having a conversation or email thread.

Become a Statistics Guru

I love statistics. I think about them night and day. You too should think about them every opportunity you have. We use statistics for everything we do as performance engineers. Think about it:

  • Response time sampling and analysis
  • Probability models for our performance simulations
  • Profiling and analysis
  • Data modeling
  • SQL tuning

The list goes on and on. If you haven’t taken a statistics class, I strongly recommend you do and do it fast. Then once you finished getting a good primer on statistics, we need you to start making recommendations on more meaningful statistics that will help the team.

The Need to Be a Generalist

Systems engineers are generalists. They need to understand hardware, software, integrations, functionality, business process, etc…For all intensive purposes, Performance Engineering is really just another name for a systems engineer. To be a great performance engineer, you need to know a lot about lot. That’s a lot to digest, but it’s true. I recommend that with each project, you try to learn as much as you can about one thing. As of late, I’ve taken this approach with the team. In Sprint 1 of 9.0 cyclical testing, the team focused on database performance. There were a lot themes such as query optimization and cost execution plans. The gist is that the team focused on just the database. In sprint 2, the team is focusing on ehCache (learning as much as they can to calibrate) and JVM instrumentation. With both projects, I can guarantee that each member of the team will learn something about their area of focus during a sprint. This way they won’t be too over loaded with data. This gets back to habit #2 (Researching). Use these sprint opportunities to focus on a given component in the stack or a particular technology as an opportunity to do external research. If you are learning about ehCache, one of the first things you should do is search for blogs that talk about ehCache on a regular or irregular basis. Chances are if someone is writing about it, they are likely to share links or references to others who are writing about the component or technology. One hint I give to you would be to look at the blogroll when you identify a good blog worth reading. I tend to build my blog OPL file based on evaluating other blogs referenced in the blog roll.

Question…Hypothesize and Prove It Out

The seventh habit is my favorite. We work in a lab. It might not be pretty and we don’t wear white overcoats. No matter how we spin it, the work we do is lab-like. That means we need to able to do 3 things really well (aside from researching). First we need to question. I don’t mean it in a negative way. I mean in it in an interrogative way. For example as a kid I always wondered how a radio worked. It’s a fairly easy thing to learn, but if you don’t start by asking questions such as: * How does a radio work when it’s not stationary? * How does it work inside buildings? From there, we go back to habit #2 and partake in research. From our research we develop hypothesis. A hypothesis is a calculated guess based on some form of empirical data. Then we must prove the hypothesis. We do that by testing. Where do we test? We test in the lab…So don’t ever forget…you work in a lab.