Looking for Ideas on How to Measure Virtual World Learning

I’m looking for some help on this. I have a couple of ideas, about measuring virtual world learning but would love to have input from others to learn how they are assessing the learning taking place in virtual worlds, please comment on your blog and then link in the comments of this blog any ideas or examples of how you are assessing learning in 3D Virtual Immersive Environments.

Here is my preliminary thinking on the topic:

Evaluating learning outcomes in a 3D Virtual Immersive Environment (VIE) should not be done in the traditional fashion of a paper and pencil multiple choice assessment. Using a multiple-choice assessment for 3D learning is like using a yard stick to measure the amount of steam produced by boiling water. You cannot possibly obtain an accurate measurement. See my discussion with Sherry Engel on this topic in her blog post Learning in 3D: Measuring the Impact.

Learning in a 3D VIE is meant to be experiential. While it is possible to craft 3D environments to teach facts and declarative knowledge, the real power of these virtual immersive environments is to allow learners to experience concepts, solve problems and work on skills like negotiation, sales acumen and team work (so called “Soft skills”).

Assessment of knowledge gained in terms of concepts, problem-solving and soft skills cannot be accurately measured with true/false or multiple choice questions. They need to be measure using different techniques such as rubrics and performance-based checklists.

Rubric

One way to construct a rubric for assessing in-world activity is to create a hierarchy of desired skills. The lowest level (and fewest points) is the minimal acceptable behavior or activity from a learner. For example, if you are teaching sales skills, it might be the opening used by the learner to begin a sale. The learner might say “Hi, my name is Juan” that might be the minimum and the learner received 5 points.

The best opening might be “Hello, my name is Juan Hertz and I am from ABC company and I’m here to talk to you today about maximizing your company’s revenue by minimizing idle inventory, do you have a few moment?” and the learner would receive 25 points.

When observing the learner role-playing in an authentic situation, the instructor assigns a skill level to the behavior and the skill level is associated with a score. This can even be done when asking learners to create content in-world. For example, you might teach a lesson on sustainability in home construction and then ask the learner to build a “green” home in a virtual world. Then the students could be assessed by the instructor walking around the green home and checking observing the sustainability element the learners included such as solar-panels, tankless hot water heater. If they included the minimum, they would receive a low score. If the integrated the elements together and carefully thought through the design, they’d receive a higher score.

The key is to design the rubric as a way of moving a learner from the state of minimal knowledge, to the state of mastery. The skill involved in creating the rubric is to understand how learners move from basic skills to the level of mastery.

Performance-based Checklist

A checklist can be used when observing the learner for specific behaviors that can be seen or not seen during an activity. Such as, “Did the officer ask to see the driver’s license?” or “Did the student check to ensure the safety guard was locked before attempting to operate the equipment?” These performances can be taught in 3D virtual spaces and then assessed either in the space as the learner runs through a simulation or, more appropriately, have the assessment occur in the physical world to note the transfer of skills and knowledge from the virtual to physical world.

Performance-based checklists can be assigned points to each item (weighted if necessary) and calculated based on how many of the items on the checklist were properly completed by the learner, they can be timed as well to see if time requirements are met.

Assessing learners in a 3D VIE should use techniques similar to assessing the learners in a physical location. You want to assess transfer of learning, ability to perform the right tasks and the ability to critically problem-solve. Two methods that help are rubrics and performance checklists but there are more.

How are you assessing learning in 3D virtual worlds, I’d love to hear some ideas?

__
Catalog of Recommended Books, Games and Gadgets
Recommended Games and Gadgets
Recommended Books
Content Guide