Grappling with an Assessment Framework for HIVEs

This blog puts forth the premise that content designed to change behavior focuses on three connected pieces: actions, results, and the often invisible system that connects them. I have probably spent more time that I should on the entries and thoughts around Actions and Systems. The notion in a sim of Results is much tougher, and I have correspondingly underwritten about them.

• Results in computer games are often simplistic, if not simple. Beat the boss or opponent. Find the key and get to the door. Build a big enough army. Finish stacking the blocks before the timer runs out.
• Some victory may be table-based. Achieve a park of size X with an income higher than Y and an average customer satisfaction of Z by time A.
• Other results may be based on a balanced scorecard methodology. Here, there may be three or four metrics that may compete for resources in the short term but all necessary in the long term.
• Students often want a single score. What is their "grade"? If they play a sim twice, how do they calibrate their relative performances between the two plays?

As is often helpful, I like to like to look at real-life as a guide. Here are some questions. How would you assess the following:

• A potential spouse?
• A job opportunity?
• A walk in the woods?
• Dinner?
• A new boss?
• A new customer?
• A diet?
• An advertising campaign?
• This morning's commute to work?
• A child's third grade experience?
• A child's college experience?
• A new chair?
• An old chair?
• A car?
• An investment?

What makes for success with each of these? When is success measured? What is the implication of relative success?

The more one thinks about it, the more confusing it can get. Does this lack of common constructs around what makes for success necessarily stunt our ability to create any formal learning program?

Measuring HIVEs

Let me up the ante just a little bit. According to my own research and others, it is become clear that we need a (better) methodology for assessing the effectiveness of Highly Interactive Virtual Environment (HIVE) Learning. For example, here are some questions:

1. Should the assessment of interactive content be interchangeable with the assessment of traditional content?

2. How do we acceptably measure Learning to Do and Learning to Be?

3. Whose data do we trust? Corporations? Schools? Vendors?

Learning Strands

I believe the answer, at least for industry metrics, will follow the framework of learning strands. Specifically, one has to prepare a program (and then assess it) much as one prepares a meal - deliberately mixing a variety of ingredients and conditions. Further, meals fit in a context. They meet goals that fall into strands: a need for calcium; a need for convenience, even portability; a need for protein; a need for flavor; a need for recognition of the chef. These strands have to be individually calibrated for the specific context.

The future of metrics and evaluations of programs may rely on their ability to meet a collection of learning strands, rather than a big, all or nothing score.