Daily Archives: July 28, 2009

Steve On SPE Voting of Clickpath Probability Models

I’m by no means attempting to be Joel Spolsky based on the title of this blog. I just wanted a simple way to say I’m really interested in talking about clickpath probability models. I think we need to consider a more effective manner of gathering input from our sprint teams with regards to our probability models that PE delivers.

Our main challenge is finding the time from others to be more interactive with our SPE efforts. Everyone is busy, so finding the time to get all of these different constitutents to provide input is darn near impossible. So maybe we have to figure out a better way to use their time more effectively for the inputs that we really need.

From an SPE perspective, the two pieces of data that we really need input from our team constituents is with regards to the lifecycle of our data model and our probability models. I think the data model discussion is a little too complex for me to start writing about here. So for simplicity purposes, I will focus on our probability models. Below is an example of a probability model taken from 9.1 about Assessment creation using question discovery.

Assessment authoring is by far one of the more critical content authoring activities in the system. It is also one of the most expensive from a resource and storage perspective. The process of authoring a question is treated with a lot TLC by content creators. How exactly do I know this? Well, I don’t have exact empirical data that says 84% of all assessment creation activity requires intense critical thinking and lasts as long as 3 hours end-to-end. Whoa…just throwing out an example. I do understand that assessments are instruments for measuring academic performance. Given that our culture is to put more utility in the measurement process, it seems likely that this would be an important activity.

I’m a victim of the problem I’ve described above, as well as a contributor. I could have easily solicited input from our team consitituents looking for their guidance about the critical nature of assessments. Some of those folks would have given an answer like mine, meaning they would believe this is a critical process given the nature of measurement in academic worlds. Some of them and hopefully the right group of people would give me the answers that they have heard first hand by the users of the system, the ones who really matter.

Getting that information is really hard. Asking the question isn’t necessarily tough. Getting the answer is tough, especially when the interviewee doesn’t have time to answer you. So I want to propose we open up our issue a little further by the team. We need to figure out how to get more precision in our probability model. Precision comes from input from our team consituents who are closer to the real data. The real data we want will shape our probability models.

As SPE engineers we should be able to read requirements and then specifications to build out all of the states, decision points and actions in order to identify the possible model paths. We should also be able to infer from the artifacts our teams give us such as the MRD and specifications our own pass at the probability models that shape the weighting of our clickpaths. What we really need is a way to receive input about these models in an easy to interpret and participate manner.

I would love for our team to rack their brain on possible solutions. At the end of our SPE day, we have to feel confident that we:

a) Receive the correct input and feedback about how users will engage with the application
b) Improve our transparency so that teams value our contributions and efforts toward the development of the product
c) Encourage our team members to consider probability models as an important component of software development

Steve’s Thoughts on Making This Happen

There’s no way that I am going to write a 2-page blog without providing my own input on a way we solicit other’s inputs. As I stated above, I think one of the challenges facing Product Managers, Designers and QA Analysts is a lack of time. For us to expect them to review our documents asynchronously is not too wise. For us to expect them to have time to meet face to face or over WebEx isn’t all that likely either. So we have to find a way to give them an easy way to review our work in a meaningful workflow that demands little time and not too much thought (as a means of reducing time and effort). They need a way to approve or question our probability models. Most importantly, we should consider building a tool set that in due time they would want to adopt themselves and ultimately seek value from (Refer to The Dip by Seth Godin).

The tool I think of is inpsired by a blog that Nakisa herself wrote back in March when she talked about a tool that’s out there on the web called Web Sequence Diagrams. This is an easy to use visualization tool that models UML interactively from simple Wiki markup and subsequently creates UML notations.

Another tool that has inspired me is Cuzillion from Steve Souders which is another simple modeling tool, but this time for modeling the ordering and sequence of web parts of an HTML page.

Both tools provide a very simple interface to construct models for their purpose. I envision that we need a tool that gives us the ability to model states, decision points and actions. We then need a way to highlight the critical paths. In essence, each path needs a weighting system to differentiate probability. We need the modeling aspect and the visualization aspect, but we also need a way to solicit input. One way is through a publish and vote process where constituents could be sent a workflow to review the probability model and then vote on its accuracy. We could also give the ability to fix the model and then compare the two or more models head to head.

If I didn’t mention it before, if we built a tool like this, it would go into Galileo (our home grown system) and be used for other data oriented projects.

As I said before and will say again, we really need input from the team about how to solve the problem of getting input. The manner in which we get the data is up for debate. Let’s start debating…