A Proposed Simulation Assessment Methodology

I have been wrestling with implementing an online educational simulation assessment methodology to prove that learning has happened during the course of a simulation to third parties. Rather than being theoretically perfect, however, to meet the needs of my clients, I needed something that works well in the field. I think I have come up with a methodology, let's call it "The Bristol Method", that might work quite well to assess a one to two hour long course.

The core assessment methodology is a simple model: A student will get sets of screens, with the task of quickly connecting boxed items from the left with the correct corresponding items on the right by drawing a line with a mouse.

The directions to the students will be as follows:

• You will get five sets of screens; in each you will connect an item from the left column with an item on the right column. You will do this by using the mouse to click on the item on the left, which will highlight, and then click on the most corresponding item on the right. A line will be drawn showing that they are connected. If you make a mistake, reclick on an item in the left column, and do the process again. There is a 90 second timer for each screen. If you are done before the time allowed, you can press the “done” button. If you finish early (before the timer runs out), you will get some bonus points.
  
• After each screen, you will see which are the right answers and a score. Your score is based on both the number correct, as well as any bonus points for hitting “done” before the timer goes off. The first of the five screens will be a practice round, which is not scored.

To launch the assessment in the first place, the student have to demonstrate a basic understanding of the connecting action, as such:

Then, again per directions, the first screen is a practice screen. For example:

This will be followed by a review screen, showing which answers are right or wrong (this is important feedback to reward students), the time left if any, and a score comprised of the two. Specifically, students get 60 points as a base, and then seven points for every right answer, plus one point for ever ten seconds left over on the clock, up to a maximum of 100 points (these points may be adjusted).

Students then have the ability to go back and replay the test screen if they want. When they are ready, the students can go through the four "real" assessment screens and sets. Each of the four real assessment screens would show five scrambled questions and answers, drawn randomly from a pool of about fifteen possible pairs. The tone of the assessment should be like a computer arcade game, with “fun” animations. At the end of the assessment they also get a final score with an average of all four sets.

The Questions

Obviously, the Bristol Method requires good questions. They can vary, although be organized by screens. So screens could include:

• Define the terms;
• Given brief overviews of situations, identify the best strategy;
• Given quotes, identify the meaning.

The test questions must line up with the learning objectives of the course.

Process

This same test (although with randomized questions each time) could be applied before and after a student went through a simulation, and can also be applied before and after a student went through a non-simulation class, as well as no class at all for baselines.

Result

If the simulation was effective, the result should show that students answered more question correctly, and in less time, and in a way that is directly comparable (and favorable) to a shift in other methods or no methods. An aggregated graph may look like:

Rational

I believe this Bristol Method would solve a handful of traditional problems.

First, it would move quickly, and be enjoyable (or less miserable than a traditional test). It could have some great bells and whistles. This is critical because any student taking a pre-test in a subject in which they don't know much can necessarily be miserable, and a harsh way to start any educational program. (Starting a two hour long simulation course having to answer questions to which a student doesn't know the answer can create defensiveness that puts a damper on the entire experience.) By matching (which rewards some knowledge by reducing the number of possibilities with each correct answer), by having a short timer (win, loose, or draw, the experience is over quickly), and by learning what they got right and wrong in the review, the experience can be as painless as possible. Further, the lower the student resistance to an assessment, the easier it is to create base cases.

Second, by measuring speed, a test can ask questions that in an untimed methodology, the student could "figure out" given more time (which are often the types of questions one wants to ask anyway.). It also minimizes the cheating options of an open-book tests. And given that an advantage of a simulation is instinctive, intrinsic knowledge, looking at speed is relevant.

Third, the content is fair and objective, and can show improvements (or lack thereof) in a way that is convincing to the outside world. And the long term time frame and low completion rates of 360 assessments, while more fair, are avoided.

Conclusion

There are a range of theoretical evaluation strategies, none of which are perfect. But I hope the Bristol Method provides authentic, reliable results, that fairly and efficiently evaluates simulation deployments. I will be testing it in months to come and let you know how well the results work.

The top five reasons why computer game designers should care about serious games

For those people getting excited about Harrisburg University of Science and Technology's Learning and Entertainment Evolution Forum (LEEF), here are the top five reasons why game designers and publishers should care about serious games:

1. Many of the most successful computer games ever, such as Roller Coaster Tycoon and SimCity, have been serious games. (See Big Skills as a game design challenge, and even Middle Skills.)
2. Serious games are making significant progress around artificial personalities, including dialogue, body language, and belief systems, that traditional computer games need. (See Creating Artificial Personalities, not necessarily Artificial Intelligence.)
3. Much as current movies are borrowing heavily from documentaries (shaky-cam, anyone?), so can computer games borrow from serious games (including virtual products) new interfaces and game-play models to add realism to experiences.
4. Serious games are developing new genres of interfaces, goals, and gameplay that can be evolved into either completely new computer games or additions to existing genres (See Genres). Because serious games designers are not trapped by the conservative design required of huge budget productions, they can explore faster.
5. Adopted and supported games used in classrooms is maturing into a long-term, stable source of revenue that circumvent the three-month hit-or-failure current computer game model. (See Top Ten Serious Games and Educational Simulations used in College Classrooms.)

Alan Kay and human universals vs. non-universals

When I was talking to Alan Kay about educational simulations a few weeks ago, he shared a model that I found helpful. Kay spoke about human universals vs. non-universals.

The universals are a list of characteristics of virtually all cultures, and certainly all children, share. These universals include:

• Social
• Language
• Culture
• Fantasies
• Stories
• Tools, Art, Technologies
• Goals, Plans ...
• Play & Games
• Fixed Rules, Flexible Strategies
• Case based learning
• Case based reasoning
• Superstition
• Religion/Magic
• Theater
• Simple, Short term fixes
• Quick Reactions To Patterns
• "The Other"
• Supernormal Responses
• Vendetta

He compared these to non-universals, which I would describe as non-intuitive perspectives but, once hard-earned, are seen as self-evident. There are examples of systems (the often invisible stuff connecting actions and results) written about here. These represent a "cultural technology," and include:

• Writing & Reading
• Deductive Abstract Math
• Model Based Science
• Thought, Thought, Thought
• Equal Rights
• Democracy
• Similarities over Differences
• Slow Deep Thinking
• Legal System over Vendetta
• Perspective Drawing
• Theory of Harmony
• Agriculture

His point was that movies and advertisements and other pop-culture tend to invoke (and pander to) the universals. Those are the easy things, the hardwired things. I am guessing all upcoming summer movies will borrow much from the first list. But in Alan Kay's perspective, education must develop conviction in the non-universals.

I think all of us designers have dipped into (sometimes heavily) the list of universals, and even included some as acceptable learning outcomes. And fairly or not, a lot of people associate games (even serious games) with the reinforcing this list of universals.

The trick may be to "pace than lead," to use the universals as a pathway to the non-universals. Students praise our design in the short term for the universals we reinforce. But they praise our content in the long term for the non-universals.

A lot of "revolutionary" thinkers about twenty years ago, like John Seely Brown, asked more of us to summon our hidden child, to challenge assumptions and unlearn our baggage. Given how many of the people in the workplace have defaulted to "winging it" (invariably with huge amount of fake/unearned confidence and even underlying threats) I may now implore more of us to nurture our hidden adult.

Podcasts for the Learning and Entertainment Evolution Forum (LEEF) on June 18-19

For those of you who like to hear an author's voice rather than read his or her words, here are some short podcasts, at http://www.harrisburgu.net/leef2009/podcast/podcast.xml.

Or:

Clark_1.mp3

Clark_2.mp3

Clark_3.mp3

These lead up to Harrisburg University of Science and Technology's Learning and Entertainment Evolution Forum (LEEF) on June 18-19, where I will be a keynoter. For more information, check out: http://www.leef2009.net/.

I hope to see you there!

What is the difference between a game and a simulation?

I am often asked "what is the difference between a game and a simulation?" I introduced a HIVE framework to suggest that Virtual Worlds, Games, and Simulations were nested concepts, better understood as discrete parts of a continuum than either as synonymous or totally separate.

The difference between game and sim is in both the media itself and the attitude and goal of the player engaging it.

Allow me to go a bit deeper. One useful analogy for a virtual world is the synthetic world of a swimming pool. So I created this chart of how the various activities one can do in a pool line up with both games and simulations.

As always, I welcome your feedback.

Clip of Japanese Version of Virtual Leader

Virtual Leader and (VLeader 2009) has been translated and deployed all over the world, proving that there is a unvirsality to many of the Big Skills written about here. Take a look at a clip of one version, created by SimuLearn's wonderful Japan partner, I-Think:

The Competition between 21st Century Skills vs. Retraining vs. Science and Math in Obama Education Priorities

I have been spending a lot of time in DC over the last month, participating in planning sessions at increasingly high levels. And I am being drawn into the interesting competition in the Obama administration between three different education objectives.

The first is to develop what is being popularly called 21st Century skills (that I have called Big Skills). These are skills that have not showed up on traditional curricula, and are around topics like leadership, project management, innovation, and stewardship. The excitement in this area is that it could challenge the traditional K-12 curricula in areas that would both help students immediately in their day-to-day life, give them more power and control of their entire lives, and also align schools with business, whose absence of such critical skills have largely resulted in the current economic crisis and US decline in global competitiveness. The problem is that plenty people believe that all of the skills are unteachable. Thus, in swinging for a home run, Obama could spend precious time and resources and whiff completely. And of course plenty of academics believe leadership (and other "learning to do" skills) is vocational.

A second contingency is focused on how to retrain American workers for what is thought to be new jobs in the new economy. They wonder, for example, what will it take to train fired car manufacturer employees into people who can install and maintain new wind turbines.

A third contingency believes that education should be increasingly focused in the traditional but underfunded and underdeveloped areas of pure science and engineering. These advocates cite recent declines in patents relative to other countries and innovation based manufacturing as proof that we need to double down, or even triple down, our efforts in these areas. The critics however might suggest that the current emphasis on science and engineering is too limited, too exclusive, and just not the right fit for too many people.

The good news for us simulation designers is that we will play a critical role in any of these three areas. We may uniquely be able to create media to support the 21st-century skill goals. We could drastically cut the costs and increase the efficiency (including scale) of a retraining focus. And we could lead the revolution in re-thinking and re-engaging a new generation in science and engineering. The only bad scenario however may be the most probable - when all is said and done, nothing new really happens.

The Need for Sleep to Process Information in a Simulation-Centric Class

Should we add "bed," alongside whiteboard and lab, in our list of great educational tools?

I have found with any experiential and complex-systems based learning program (such as around a Big Skill like project management), it is paramount to have the participants first get an exposure to the task, and then "sleep on it" before continuing. When students were not able to sleep on it, they were anxious and dissatisfied and learned less. In contrast, when the students did break up learning with sleep, their subconscious processed and assimilated the information, and they returned to the program the next morning without the trepidation they had shown the night before and in the control groups.

Said simply, the same program that took the same number of hours, if broken up with a good night's sleep, resulted in significantly better student enjoyment and, more importantly, organization and retention of the material.

Possible problems of ignoring the role of sleep

The existance of this simple rule can hurt simulation deployments in at least three different ways. First, this can confound a training group's insistence on a "one-day" or "half-day" program, especially where students are unreliable in doing any prework (universities, thankfully, don't have this problem). This also can hurt some attempts to measure the effectiveness of simulations, as researchers often try to control all variables and shoe-horn in an entire simulation experience in a single (often long) session. Finally, this can hurt the widespread adoption of a simulation if an evaluator tries to skim a simulation in a half-hour, and then "doesn't get it" so doesn't support it.

Chunking well

As with a fine wine, authentic learning has to breathe a bit. A simple chunking process, where students experience at least 30 minutes to an hour of the interface in its entirety and at least some of the mechanics, even ideally getting a little stuck (which can be done as homework if the students are responsible and the deployers of the class have credibility), sleep on it, and then dive in to harder levels can be the difference between success and failure, between meaningful experience and frustration and confusion.