Monthly Archives: July 2009

When Did This Java Issue Creep Up on Windows

I normally wouldn’t post something like this, but I think it’s weird that we are seeing this problem…plus I already posted the issue on Sun’s Java Forums.

Yesterday I ran into Mesfin as I was heading to the eye doctor. He mentioned that he was unable to start-up his developer build when his -Xmx parameter was set above 1024m. He said that other teammates reported the same issue on their Windows devpods. I sat with him at his machine for a few minutes to figure out what’s up. There’s a couple of things we know:

  • Java on 32-bit Windows should support around a 1.7GB heap
    • In Mesfin’s case, he was starting a VM with 1250mb + 256 MaxPerm + 70mb in stack memory, so it was clearly well below 1.7GB
    • He could start up using 1024mb without issue, so it appears there’s an artificial barrier of around 1.3GB
  • I recently had Windows Updates applied on both W2k3 Server and XP
    • Apparently in the past there have been issues of these heap problems after Windows Updates.
    • Someone else wrote about this a few months back, but did not come to a resolution as to why.
  • Our stack size recently was changed to 320kb, up from 160kb
    • I don’t think this is really the issue as I’ve dropped the stack size down a bunch.
    • Only solution for keeping a 1250mb heap was to drop the PermSize below 10mb, though it was totally unscientific and more analysis between 256mb and 10mb would need to be explored.

So I decided to rebuild my machine with the latest and greatest mainline. I used the following parameters below with all of the other defaults with the product.

# Initial Java Heap Size

# Maximum Java Heap Size
I received the following exception:
INFO   | jvm 1    | 2009/07/28 13:39:12 | Error occurred during initialization of VM
INFO   | jvm 1    | 2009/07/28 13:39:12 | Could not reserve enough space for object heap
INFO   | jvm 1    | 2009/07/28 13:39:12 | Could not create the Java virtual machine.
ERROR  | wrapper  | 2009/07/28 13:39:12 | JVM exited while loading the application.

What’s crazy is that we have been doing tests non-stop with values above 1024m for months. So why are we seeing this now? Why hasn’t anyone else seen this issue out on the web (recently that is)? Why haven’t any customers reported this issue? If they used the tuning set from 9.0 SP1 on 32-bit they would have experienced it first hand.

I’m going to do a little more research. Curious if anyone has thoughts…

Back From Some Research

OK…so this is messed up. I tried launching the JVM with 1mb of PermSize and heap sizes of 1.4g, 1.2g, 1.1g, 1050mb and even 1025mb. Yes that’s right, 1mb more then 1GB and guess what happened? I’m not kidding you when I say I could not start the heap.

I then wanted to see what would happen if I used an MX of 1024m, but a PermSize larger. I tried PermSizes of 600m, 512m, 300m and 256m. The only one that would start without a heap reservation issue was the 256mb.

So we have a problem to say the least. It’s clearly not an application issue, but either a JDK issue or a Windows issue. Either way, the best folks to resolve this would be Sun. I’m going to submit a ticket and see what happens. In the meantime, I’m going to make support and engineering services aware of this issue.

Steve On SPE Voting of Clickpath Probability Models

I’m by no means attempting to be Joel Spolsky based on the title of this blog. I just wanted a simple way to say I’m really interested in talking about clickpath probability models. I think we need to consider a more effective manner of gathering input from our sprint teams with regards to our probability models that PE delivers.

Our main challenge is finding the time from others to be more interactive with our SPE efforts. Everyone is busy, so finding the time to get all of these different constitutents to provide input is darn near impossible. So maybe we have to figure out a better way to use their time more effectively for the inputs that we really need.

From an SPE perspective, the two pieces of data that we really need input from our team constituents is with regards to the lifecycle of our data model and our probability models. I think the data model discussion is a little too complex for me to start writing about here. So for simplicity purposes, I will focus on our probability models. Below is an example of a probability model taken from 9.1 about Assessment creation using question discovery.

Assessment authoring is by far one of the more critical content authoring activities in the system. It is also one of the most expensive from a resource and storage perspective. The process of authoring a question is treated with a lot TLC by content creators. How exactly do I know this? Well, I don’t have exact empirical data that says 84% of all assessment creation activity requires intense critical thinking and lasts as long as 3 hours end-to-end. Whoa…just throwing out an example. I do understand that assessments are instruments for measuring academic performance. Given that our culture is to put more utility in the measurement process, it seems likely that this would be an important activity.

I’m a victim of the problem I’ve described above, as well as a contributor. I could have easily solicited input from our team consitituents looking for their guidance about the critical nature of assessments. Some of those folks would have given an answer like mine, meaning they would believe this is a critical process given the nature of measurement in academic worlds. Some of them and hopefully the right group of people would give me the answers that they have heard first hand by the users of the system, the ones who really matter.

Getting that information is really hard. Asking the question isn’t necessarily tough. Getting the answer is tough, especially when the interviewee doesn’t have time to answer you. So I want to propose we open up our issue a little further by the team. We need to figure out how to get more precision in our probability model. Precision comes from input from our team consituents who are closer to the real data. The real data we want will shape our probability models.

As SPE engineers we should be able to read requirements and then specifications to build out all of the states, decision points and actions in order to identify the possible model paths. We should also be able to infer from the artifacts our teams give us such as the MRD and specifications our own pass at the probability models that shape the weighting of our clickpaths. What we really need is a way to receive input about these models in an easy to interpret and participate manner.

I would love for our team to rack their brain on possible solutions. At the end of our SPE day, we have to feel confident that we:

a) Receive the correct input and feedback about how users will engage with the application
b) Improve our transparency so that teams value our contributions and efforts toward the development of the product
c) Encourage our team members to consider probability models as an important component of software development

Steve’s Thoughts on Making This Happen

There’s no way that I am going to write a 2-page blog without providing my own input on a way we solicit other’s inputs. As I stated above, I think one of the challenges facing Product Managers, Designers and QA Analysts is a lack of time. For us to expect them to review our documents asynchronously is not too wise. For us to expect them to have time to meet face to face or over WebEx isn’t all that likely either. So we have to find a way to give them an easy way to review our work in a meaningful workflow that demands little time and not too much thought (as a means of reducing time and effort). They need a way to approve or question our probability models. Most importantly, we should consider building a tool set that in due time they would want to adopt themselves and ultimately seek value from (Refer to The Dip by Seth Godin).

The tool I think of is inpsired by a blog that Nakisa herself wrote back in March when she talked about a tool that’s out there on the web called Web Sequence Diagrams. This is an easy to use visualization tool that models UML interactively from simple Wiki markup and subsequently creates UML notations.

Another tool that has inspired me is Cuzillion from Steve Souders which is another simple modeling tool, but this time for modeling the ordering and sequence of web parts of an HTML page.

Both tools provide a very simple interface to construct models for their purpose. I envision that we need a tool that gives us the ability to model states, decision points and actions. We then need a way to highlight the critical paths. In essence, each path needs a weighting system to differentiate probability. We need the modeling aspect and the visualization aspect, but we also need a way to solicit input. One way is through a publish and vote process where constituents could be sent a workflow to review the probability model and then vote on its accuracy. We could also give the ability to fix the model and then compare the two or more models head to head.

If I didn’t mention it before, if we built a tool like this, it would go into Galileo (our home grown system) and be used for other data oriented projects.

As I said before and will say again, we really need input from the team about how to solve the problem of getting input. The manner in which we get the data is up for debate. Let’s start debating…

The Sixth Sense of Benchmarking

A few days ago I was having a conversation with some folks about the unlikeliness of a movie like the Sixth Sense or even the Usual Suspects working in today’s connected society. People would twitter, post to facebook or blog the outcomes of those movies within minutes. They would be leaked over YouTube or covered in US Weekly. It’s kind of bummer because those were two awesome movies. From my point of view, they worked and I would love to see more movies like them in the future.

This got me thinking about what it means to have a Sixth Sense in the world of benchmarking. Is there some form of clairvoyance that performance engineers should have in general? I think so, which is why I’m writing this blog. Performance engineers should constantly be thinking about the line that exists between the software execution model and the system execution model. The software execution model characterizes the resource requirements of the software alone, without considering other workloads, multiple users, or delays due to resource contention. The system execution model characterizes the performance of the software in the presence of factors which could cause contention for resources.

I tend to think about the software execution model in terms of focus tests and isolated design of experiments. I think of the system execution model in terms of design of experiment, but mainly from the perspective of system components. The system execution model is what benchmarks are all about. So what exactly is this Sixth Sense I reference, but don’t actually define?

Benchmarking is about telling stories. Any given benchmark might contain 1, 2, 3 or even 10 stories. The more interconnected the stories, the better the benchmark. Performance engineers have to be able to seek out the story lines, find the connections between the stories and figure out a way to present combined stories as one big story. So the Sixth Sense is the ability to tell a really good benchmarking story.

Our team just came back from our most successful benchmark ever in the history of our team. We had a number of story lines around platform technologies, as well as a series of stories with regards to behavior and adoption of our system. We wanted to study the system execution model for an institution primarily using Blackboard as a distance learning tool. That meant a 100% online interaction model for everything from course delivery to collaboration to assessment. Sessions would be longer. Clicks would be deeper. Most importantly a larger percentage of users would be concurrently active in the system at any given time.

We had a lot of technology story lines to study as well. One story was around identifying the best set of standard (-X) and non-standard (-XX) JVM options that would reduce our latency introduced at the application layer. There was a subplot about JVM sizes in a 64-bit stack. Was there a big difference between a 2GB vs. 4GB vs. 8GB vs. 16GB JVM? Another story was constructed about memory management in Oracle 11G. Of course we had a great story line about virtualization using Citrix XenServer with regards to identifying our optimal collapse ratios.

There are a lot of good stories available for us with our current release. I’ve seen one good story thus far in which the team would like to compare the performance and scalability of virtualization versus clustering as a deployment strategy. I would love to see a story about SQL Server 2008 from a scalability perspective. A lot has been written about performance and scalability of the platform. There’s also a webcast worth listening. We should have dozens of questions about the latest technology. Though I’m partial to always investigating the query optimizer as my first initiative when it comes to an updated database.

These are just a few of the questions we have to ask as we prepare benchmarks going forward. Coming up with questions is the best way to build out plots which ultimately make our stories.

BbWorld09 In the Books

I just took a nice little 3 hour nap. Boy did that feel good after a long, but very enjoyable week with colleagues. I gave my last session this morning to a small intimate group of fellow Performance Tribe members. I really enjoyed the last session the most of all. It just seemed like everyone came together to work with each other on their performance and scalability issues. I also got to meet some great folks from the ANGEL community from Penn State.

I’ve attached the slides from the session. Hope everyone enjoys…

See you next year at BbWorld next year in Orlando.

Happy Reading!


Slides from BbWorld 09 Main Conference Forensics Presentation

Hi readers,

Attached are links to the forensics presentation and a sample neXpert report. Thanks for everyone who came to the session. For those who couldn’t because of the Sys Admin group meeting, well you missed a good one 😉

Enjoy the slides!

1. Main Presentation

2. neXpert Report


Slides from Optimizing BbLearn at BbWorld09

Thanks again to everyone who came to the session. There were some great questions asked. I’ve posted the slides that are really to accompany the paper on Behind the Blackboard. I’ve attached the slides to SlideShare.

Don’t forget I will be presenting tomorrow on Software Performance Forensics and then again on Thursday on scaling really large Blackboard deployments. Hope to see you in both sessions…


Steve Feldman

Pre-Con Presentation #2 in the Books…Main Conference Here We Come

The first 5 hours of presentations are complete for me. This morning I co-presented with Bryan Wynns of Quest Software about the Reference Architecture and how we implement monitoring and management. I’ve put the slides on SlideShare here. Feel free to access them.

Overall the presentation went well. Numbers were down from previous preconference presentations. For those 14 people who came to the presentation, I hope we gave them a good 2 and 1/2 hours of viewing enjoyment.

For the first time in quite a while I started watching the Twitter feed. There’s been a little activity on the Twitter about my presentations and this blog. I’ve gotten more traffic over the past 2 days then I got all of this year. Granted I took a 6-month break from writing.

Thanks again from coming for those who attended…for those who couldn’t make it, I hope to see you at the main conference!