just drafts

According to a 2011 metastudy by Traci Sitzmann in Personnel Psychology, declarative and procedural knowledge and retention were observed to be higher in groups taught with computer-based simulation games than in groups taught without, and even self-efficacy was observed to be substantially higher—surprisingly high, I might say. But that isn’t the whole story.

Common knowledge, and often among the main rationales for developing computer-based simulation games, is that wrapping entertainment around course materials will boost motivation. Motivation, hopefully, for learning new skills and not merely for playing the simulation game.

But do we know for sure that this works?

Two key simulation game theories propose that the primary benefit of using simulation games in training is their motivational potential. Thus, it is ironic that a dearth of research has compared posttraining motivation for trainees taught with simulation games to a comparison group. A number of studies have compared changes in motivation and other affective outcomes from pre- to posttraining for trainees taught with simulation games, but this research design suffers from numerous internal validity threats, including history, selection, and maturation. Also, the use of pre-to-post comparisons may result in an upward bias in effect sizes, leading researchers to overestimate the effect of simulation games on motivational processes.

Sounds bad enough. But there’s more! In a corporate environment, motivation is intimately linked to work-motivation—think of it as a special case of transfer of learning—but which, it turns out, hasn’t so far been tested in any meaningful manner at all:

However, the instructional benefits of simulation games would be maximized if trainees were also motivated to utilize the knowledge and skills taught in simulation games on the job. Confirming that simulation games enhance work-related motivation is a critical area for future research.

Also, there’s something else. How well declarative and procedural knowledge, retention, and self-efficacy are raised depends, according to this meta analysis, on several factors. The best results were observed for games where work-related competencies were actively rather than passively learned during game play; when the game could be played as often as desired; and when the simulation game was embedded in an instructional program rather than a stand-alone device.

Lots of implications there. And ample opportunity to turn your corporate simulation game into a veritable shit sandwich: when the game is merely the digital version of your textbooks, training handbooks, or field guides; when the replay value is low; and when you think you can cut down on your programs, trainers, and field exercises.

In other words: a good simulation game will cost you, and you can’t recover these costs by cutting down on your training environment. Instead, a simulation game is a substantial investment in your internal market, and you better make sure to get the right team on board so that motivation will translate into training success and training success into work-motivation.

Paper cited: Sitzmann, Tracy. “A Meta-Analytic Examination of the Instruction Effectiveness of Computer-Based Simulation Games.” Personnel Psychology. Vol.64, Issue 2 (Summer 2011). 489–528.

How serious games are developed has changed quite a bit since Gunter et al.’s paper “A Case for a Formal Design Paradigm for Serious Games” (link to PDF) from 2006, but that doesn’t invalidate its point of departure in principle:

We are witnessing a mad rush to pour educational content into games or to use games in the classroom in an inappropriate manner and in an ad hoc manner in hopes that players are motivated to learn simply because the content is housed inside a game.

While this paper is neither a rigorously written research study nor exactly informed by deep knowledge about the psychology of learning (all three authors have their backgrounds in the technology of learning), and the concluding “method for creating designed choices” falls flat on its nose as this paper regrettably fails to define “choice” in this context, we can still extract its basic idea, strip off its naïve linearity, and expand on it.

In brief:

The basic design process for educational games should occur within a three-dimensional space whose three conceptual axes are: Game Mechanics, Dramatic Structure, and the Psychology of Learning. To simply try and “map” these parameters onto each other in a largely linear approach that, among other things, is destined to lose sight of participatory elements and agenticity rather quickly will run into problems and lead to bad games. And the best approach to build such a matrix for a given objective is to create a collaborative team with top-notch professionals from all three areas, i. e., game design, narrative design, and the psychology of learning and motivation.

Paper cited: Gunter, Glenda A., Robert F. Kenny, & Erik Henry Vick. “A Case for a Formal Design Paradigm for Serious Games.” The Journal of the International Digital Media and Arts Association. Vol.3 No.1 (2006). 1-19.

From a 2013 Research Roundup on Game-Based Learning:

While serious games have been embraced by educators in and out of the classroom, many questions remain. What are the possible effects of digital gaming, connectivity and multitasking for younger learners, whose bodies and brains are still maturing?

Let me rephrase this just a bit:

While 20th century-style classroom learning has been embraced by educators all over the world, many questions remain. What are the possible effects of one-size-fits-all educational methodology with predetermined curricula and standardized testing, conditioned learning of siloed educational subjects detached from personal experience, and large class sizes solely determined by year of birth, for younger learners whose bodies and brains are still maturing?

What this comes down to is this. With their defensive positions reflected by arguments as well as study designs, game-based learning proponents often paint themselves into a corner. You just can’t conclusively identify (let alone “prove”) the effects and effect sizes of a particular teaching method for all times, ages, and contexts. Moreover, it’s proponents of that archaic industrial processing of learning and learners that we, somewhat misleadingly, call our “modern educational system” who should scramble to legitimite their adherence to outdated structures and methods, not the other way round.

Another thing that’s screwed, of course, is that from twenty studies on game based-learning listed by this particular research roundup mentioned above, only three are freely available — “Video Game–Based Learning: An Emerging Paradigm for Instruction” (Link); “Gamification in a Social Learning Environment” (Link to PDF); “A Meta-Analytic Examination of the Instructional Effectiveness of Computer-Based Simulation Games” (Link). And from the other seventeen arcticles’ overall ten sources even the excellently-equipped university and state library I’m privileged to enjoy research access to does subscribe, again, to three.

A circulating story about a grey parrot performing unexpectedly well in a setup of the classic 1972 Stanford Marshmallow Experiment reminded me of a related study from 2012, “Rational Snacking: Young Children’s Decision-Making on the Marshmallow Task Is Moderated by Beliefs About Environmental Reliability” (paywall). Takeaways from the latter were: the original test’s setup didn’t control for trust (and probably other factors as well), and the dependent variable of self-control is at least “moderated” by the perceived reliability of the environment—trust—in personal delay-of-gratification success or failure. (About the parrot, alas, I have nothing useful to say.)

One can be reasonably sure that both factors, i.e., trust and self-control, are indeed highly correlated in real-world settings. From there we can assume another confounding factor pertaining to both the original and the follow-up study that hasn’t been controlled for, namely, social background. Reliability isn’t just a matter of character, but also a matter of circumstances: people can be unreliable und untrustworthy not because they are unreliable and untrustworthy, but also because their personal environment and circumstances force them to be. For a taste of how this works, play the terrific Papers, Please!.

In fact, your unreliability and untrustworthiness in real-world settings can be a function of naked necessity, and the more precarious your personal circumstances are, the less you can afford being reliable and trustworthy. It can start with the promise of a birthday present you couldn’t keep because you needed to have the refrigerator repaired that just broke down, or for a visit to the zoo or your daughter’s all-important afternoon baseball game because you finally didn’t dare leave early from work even though you had asked for permission.

Now think again about the results of the original Marshmallow Experiment’s follow-up studies:

In follow-up studies, Mischel found unexpected correlations between the results of the marshmallow test and the success of the children many years later. The first follow-up study, in 1988, showed that “preschool children who delayed gratification longer in the self-imposed delay paradigm, were described more than 10 years later by their parents as adolescents who were significantly more competent.”

What needs to be done, actually, is factoring in the participants real-life environment reliability which, stunningly, hasn’t been controlled for throughout these studies.

Now, learning to trust or not to trust is a bit more complicated than “learning whom to trust or not” as an educational objective. “Trust” has a context, often a dynamic one. If children don’t understand this, they will have no clue what to do or what’s happening, e. g., when it comes to intermittent trustworthiness because circumstances compel a basically trustworthy person to be trustworthy most of the time, but not all of the time. And we know from Behaviorism that intermittent rewards (i.e, reinforcements) trigger certain classes of hormones that are not only highly addictive, but change behavior in unexpected and overwhelmingly socially negative ways.

When “Trust” as an educational objective has to take dynamically changing contexts into account, many of the game mechanics used in educational games are not even close to being useful, and that applies to corporate trainings as well. But how can we create dynamic contexts?

Well, Storification! A dramatic structure, turning points, ever higher stakes, unforeseen predicaments, and a “story world” where every action has its consequences, though not necessarily the expected ones. Obviously, in an interactive, participative game where the player can expect to have agency it’s neither possible nor desirable to “script” all these elements beforehand. As of now, the single best way to experience such situations, and experience them in perfect safety!, are still old-fashioned pen-and-paper roleplaying games—true, authentic “co-op” games through-and-through.

And that’s exactly what we’re going to have to create in the field of game-based learning: a plot-driven, context-rich co-op game that dynamically evolves through interactions between players who enjoy true agency. Which would be almost too easy with Strong AI as a “game master,” but as long as we can’t have that, we must work with what we have. Which, actually, is quite a lot—with MMORPG mechanics and mixed AI/Tutor systems leading the way.

just drafts is one part newsticker with commentary on game industry matters, and one part regular blog about game-based learning, education in general, and LLM/GPT/AI research in particular. Regular posts on other game-related topics you’ll find at my flagship blog between drafts under the categories collateral tales (narrative) or the one and twenty (everything else).

News ticker posts will feature quotes—like this—from an external source, only longer.

To these I will add commentary. The post’s headline will link directly to the source of the quote (a pattern I picked up from John Gruber’s Daring Fireball site), indicated by an arrow at the end; a blue permalink for the post itself will be provided at the end of the post (like below). Headlines from regular posts, in contrast, will have no arrow and serve as permalinks, just like in good, old-fashioned blogposts.

Yee-haw! See y’all soon.