I was reading this paper about the differences between software development in general and game development and the authors made some good points regarding software testing, pointing out, for instance, that

...game developers are hesitant to use automated testing because of these tests' rapid obsolescence in the face of shifting creative desires of game designers.

So, this reading made me think, what other aspects in software testing should we consider as different or particular when we are dealing with/testing a game? Does anyone have experience with this or had anyone heard something else about it?

  • Mind linking to the paper? I’d be curious to read it. – RubberDuck Jan 11 '18 at 2:10
  • 1
    Here is the paper: microsoft.com/en-us/research/wp-content/uploads/2016/02/…. Oh, and give your opinion about this if you don't mind. Thanks. :-) – Ronnie Edson Jan 11 '18 at 2:41
  • 9
    I'm afraid that rapid obsolescence (of tests) in the face of shifting desires of the powers that be occurs in non-game development as well. Which suggests that perhaps game development is not that different from other development? – Erik Eidt Jan 11 '18 at 2:59
  • I would say the biggest difference between business software and games isn't "shifting requirements", which is common virtually everywhere, but the emphasis on performance and intense UI work that makes up a game. In business software, the data and logic models tend to be separated from presentation, making them easy candidates for unit testing. Games don't always have this luxury. This isn't to say that the server-side portion of online games can't be tested in more traditional ways, likewise pure game logic, monster spawning rates, etc. – Dan1701 Jan 11 '18 at 4:46
  • 1
    Different is a very broad church. And it rather depends on what you're comparing it to. – Robbie Dee Jan 11 '18 at 14:21

Modern games are actually a ton of creative art content developed using an in-house or proprietary game engine. The engine itself is unit testable for most part (rendering, geometry, physics, AI modules etc). Similarly, simple tests can also be attached to individual parts of developed content. This means unit and white-box testing is indeed feasible and successful.

As far as "product as a whole" is concerned, a game is a simulation. It might have more generative complexity than a simple business program. Think of endless, unique, procedural generated worlds versus an enterprise resource planner with countable well planned behaviors. Simply put, the number of possible unique ways to do something in context of games, can be mathematically, very very large. In fact it is considered a selling point for games.

Add to that the fact that the end output is purely audio visual and there is no deterministic standard of absolute correctness of such output. GPU chips really don't need to perform precise calculations, just a lot of calculations, even if some are not precise.

And finally, the main goal is Entertainment. Gamers are okay with glitches if it runs 60+ FPS, looks awesome, and has endless hours of entertaining content.

This simply puts the traditional automated black-box testing ideas in "not so tangible and worth it" region when applied to games.

However, there have been recent attempts to train NNs to play games, which is effectively a form of exploratory, self-learning monkey testing.

  • 2
    What is the "average business program"? – whatsisname Jan 11 '18 at 7:12
  • 1
    Yes ! It's not so much the number of interaction that is different (take a leading ERP with several thousands of interrelated transaction types and a process landscape that can be reconfigured endlessly). It's more that a business software is expected to provide a repeatable behavior that can be easily verified in an integration test. Games have to entertain and anything repeatable is boring. So it's difficult for the testing tool to measure the degree of entertainment or the consistency and realism of the scenes the user see. May be with some AI in 30 years from now....? – Christophe Jan 11 '18 at 7:57
  • @Christophe it depends on the scope of repeatable - e.g. "when the character gets shot, he should lose 5 health" is perfectly repeatable and perfectly testable. What matters is that the repeatable testable game logic is well-abstracted from the parts with less tangible states to assert against. – Ant P Jan 17 '18 at 15:32

It's been many years since I did gamedev but on top of the nice answer, there are some things I want to add and detail.

First already mentioned is that the output is just visual and auditory against tight "FPS-critical" constraints and computational/memory budgets. Ideas of correctness get blurry when the questions are more like, "Does it look good? Does it run smoothly without any stutters? Does it sound great?" while developers are tweaking and tuning and approximating while designer/dev collaborations lead to things looking and sounding slightly different with every rapid iteration.

Another one is that the testers can be awesome! I've never found a more dedicated group of testers in any other domain, since they want to test the software. They're having fun. They're addicted and sleeping next to the computer while exploring every nook and cranny of your game. It becomes pretty easy to discover even the obscurest glitches when people are actually entertained thoroughly testing every corner of the software while practically addicted to it. In my current industry the testers are a bit more difficult to work with since many of them are professionals tying their livelihood to the software, and so they rely on a handful of features to get their work done and aren't necessarily that interested in exhausting every nook and cranny all the time. Naturally when we can't rely as heavily on human testers, we need more automated testing.

Yet another is that the codebase for a game typically isn't maintained and modified and extended for years and years. It's not like the developers of Super Mario who originally developed it in 6502 assembly have had to maintain anything resembling that original code long after the game shipped. Doom 3 probably uses zero lines of code (or close) from Doom 1. If there's a continuing franchise, the newer games are more like "sequels" than "upgrades". Most games just ship and maybe release some patches, DLCs, and then the code is done. That's a huge contrast from my VFX industry where I've worked on maintaining code dating back to Amiga days which had been ported and maintained for decades. Games typically don't have 30-year old codebases let alone decade-old codebases still being maintained and aggressively changed today.

One of the reasons for this short-lived nature of game codebases is that they are so tied to the hardware. When combined with their cutting-edge nature and FPS-critical requirements, they often can't be developed in a way that abstracts hardware details, not even close. They're often written very specifically for the target generation of hardware, and it's usually not long before that PS3 becomes replaced by a PS4 which then becomes obsolete and replaced by a PS5, and so forth, and all very rapidly. The hardware capabilities play such a pivotal role in the game's design and development that it's generally not worth trying to maintain a lot of the same code written for PSX as for PS4, e.g. Most game franchises which last for generations still write their next-gen engines largely from ground-up for the newest hardware.

With a short-lived codebase comes limited maintenance time (i.e., a limited time in which the code has to be modified). With a limited time for the code to be changed that doesn't span years with the scope of the engine growing larger and larger with each upgrade, and combined with the fact that games are nowhere close to mission-critical, there's not such an absolutely critical need to apply the most exhaustive unit and integration testing. There's no benefit to doing that in ensuring the integrity of future changes if future changes aren't going to be made, and the unit testing and refactoring aspect of legacy codebases is naturally irrelevant if there's no "legacy" in the first place.

Another small one that isn't always relevant is that a game might only target a very narrow range of hardware without any desktop ports. In those cases a huge source of unpredictable glitches in these contexts, which is users running the software with radically different hardware and drivers, is eliminated.

That said, integration testing at the highest/coarsest-level tends to be more immediately useful. For example, many games might utilize a way to record how the game state is changing over time for "replays". Such replay features can ensure that the game is deterministic and also be used as a form of a testing tool on its own to replay back a game session recorded previously by someone else.

I've also encountered gamedevs working in small studios who did things like write bots for their game and had the bots play their game at max speed and ran that simulation, originally encountering an obscure crash after a day or two, then fixed it, then ran the simulation again, and repeated until there were no more show-stopping crashes even after running it for weeks on end. So there are interesting kinds of pragmatic approaches like that which I've seen from gamedevs to testing their software, but often in ways that resemble the coarsest level of integration testing and simulating things very closely to how the players actually interact with the game.

Finally these big AAA game engines are starting to resemble a whole different kind of beast: longer-lived, successfully abstracting the hardware a little better, with bigger codebases and longer maintenance spans while their level editors are starting to resemble full-blown development environments. I imagine those big engines would probably call for more thorough testing procedure, especially if the time their code is maintained expands considerably. Still a lot of game studios don't write huge AAA game engines: they either license them or develop a small proprietary engine which is considerably smaller in scope and isn't going to be maintained for years.

  • 1
    Bots. Yep, that's a tried and tested approach. – S.D. Jan 17 '18 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.