It’s not uncommon for me to get asked on Twitter (when I was active there) why we don’t unit test games? They would mention how they work in blah blah industry and do unit testing all the time and it’s virtually eliminated the need for human testers. They then go on to call me a jerk face and tell me I make stupid games.
OK, I made up that last part, but sometimes it feels like they said it.
It does bring up an interesting question. Why don’t we unit test games like you would other software? I can’t speak for others, but for me, it just isn’t that useful.
What is unit testing you ask? It’s basically where you write programs that feed data (sometimes random data) into program’s functions or modules and make sure you get the expected results or proper errors. That’s an oversimplification, but you get the idea.
Unit testing texture or asset loading can be useful, as is unit testing many other backend engine routines like memory managers, but that’s just not where most (and by that I mean 99%) of the bugs comes from.
Most of the bugs in a game (or at least my games) come from users doing stuff, and not stupid stuff, but normal expected stuff and the game logic just being wrong. This is really really hard to unit test and the failure states are hard to identify.
For Thimbleweed Park, we had the TesterTron 3000™ that randomly played the game for hours on end. This was useful in that it found occasional crash bugs, but it wasn’t very good at finding game logic bugs because, after an overnight run, it was hard to spot that something had gone wrong 6 hours ago. This is especially hard if the logic issue was very room-local and hard to spot later.
TesterTron had a mode where it would jump into each room one-by-one and this was useful in catching missing assets or bad packaging.
But we encounter a lot of bugs where the game works fine, but something is just visually wrong. The Sheriff picks up the phone and his animation isn’t aligned correctly. Or worse, it’s not aligned correctly only after he’s also open the jail door. Or Delores stands in the just the right spot and she clips into the desk.
I just don’t know how to realistically unit test this stuff. It’s possible, but the cost and effort involved would quickly outweigh the benefits. We’re not making flight software for 777s. No one of going to die or lose their entire bank balance because of our bugs. We’re not writing enterprise level software, we’re making video games and the cost of most of our bugs are negligible compared to monitoring a nuclear reactor.
It’s not that we don’t care, we care a lot. One of the largest expenses in Thimbleweed Park were testers.
I say “most of our bugs”, but not all. Post Thimbleweed Park launch, we did discover some bugs that caused players to lose progress and these are very painful to us.
We fix them as fast as possible and release a new build, but with each of them, I don’t know how unit testing would have helped. We had a problem on the PS4 where savegame data would get corrupt due to threading. We could have built a stress tester for this, but that’s all 20/20 hindsight. I can think of 100 other places that worked fine and without that hindsight, we would have spent a lot of time writing stress testers.
Again, not impossible, just not economical.
But maybe I’m wrong. Maybe other game developers have amazing game unit tests that are enormous time and money savers. If that’s true, I’d love to hear about them. I’m basically lazy and more than happy for robots to do my work.
But it feels like the ability to unit test games would require a good AI that can spot perceptual issues that go way beyond just getting bad data out of a function. Or maybe that’s called Early Access.
