Commandments of Quality Assurance

by Chuck McFadden

1. Employ the Scientific Method.
2. Understand the difference between playing a game and testing a game. Spend most of your time doing the latter.
3. Be flexible.
4. Find and report bugs as early as possible.
5. Think like a hacker. Be creative in finding problems with the game.
6. Put in as much effort with your regression testing as you do with your initial testing.
7. Don't let Q/A members test designs they've created.
8. Don't write sloppy bugs. Spell and grammar check everything.
9. Test everything you can reasonably test.
10. Work under the assumption that most (if not all) bugs can be consistently reproduced.

IF Q/A HAD A SET OF 10 COMMANDMENTS, WHAT WOULD THEY BE?

Approximately two years ago, a member of the IGDA's Q/A Special InterestGroup [SIG] posed that very question, which ignited the SIG into lengthy discussions and friendly arguments until, weeks later, we settled on 10. They were compiled and listed on the SIG site, where they have remained in relative obscurity... until now.
This article lists the commandments in no particular order and examines each in detail. Each has equal value; no one commandment is more important thanany other. But ignore them and the god of game development will smite you.

1. EMPLOY THE SCIENTIFIC METHOD.

When you boil it down, this is what separates a good tester froma bad tester: the good tester—consciously or unconsciously—uses the Scientific Method. Make this a part of your Q/A team's training process, just in case one of your testers paid as little attention during science class as this author. If trained well in the Scientific Method, upon observing a bug, a tester will:

a) Observe and describe. At the least, the tester should take notes on what happened. Ideally, your tester is recording everything (with a VCR or digitally), which helps the tester with the next few steps. Either way, this step simply states what happened, for example that the game crashed at the second save point.

b) Formulate a hypothesis. The tester should speculate on what might have caused the bug. Did they try to save when the game was streaming off the disc? Was a character talking when the save action triggered?
Did the character "die" at the sametime the save initialized? Based on the observed behavior, a tester can develop a testable hypothesis to repeat the bug. (An inexperienced or less-thorough tester will end the process here, writing up the bug based on his/her speculation.)

c) Experiment. The hypothesis should then be tested to see if it results in the initially observed behavior/bug. Re-experiment to narrow down the steps to repeat the bug. If the hypothesis is true, move onto the next step. If the hypothesis is false, refer to step A and start again at step B.

d) Draw a conclusion and communicate the results. Once the tester has verified that they can repeat the bug with the minimal number of steps (in other words, once they have verified that the hypothesis is true), they should write it up. How your tester writes up the bug is up to you and your established procedure. Regardless of how they write up the bug, if they use this method, the bug will be solidly researched and consistently reproducible.

2. UNDERSTAND THE DIFFERENCE BETWEEN PLAYING A GAME AND TESTING A GAME. SPEND MOST OF YOUR TIME DOING THE LATTER.

The most common misconception I've seen with new hires (and with industry colleagues who don't know much about Q/A) is they believe testers "play games all day." If this is 100 percenttrue with your Q/A team, you release terribly buggy games. Quality assurance is not just a matter of testing how a game issupposed to work, but how the game is not supposed to work.

The difference is simple. A good tester knows that losingneeds to be tested as thoroughly as winning. He understandsthat getting the best lap time in a racing game is only half thejob. Otherwise, a bug as simple as a crash at the "game over"screen will never come up if the tester is always playing to win.

This commandment also speaks to another philosophy: Don't exclusively hire "hardcore" gamers. While employing hardcore gamers is important, it's equally important to cultivate more casual players. As a lead or manager, you'll be tempted to hire the most dedicated gamers in your neighborhood. Unfortunately those testers invariably have the hardest time losing. Moreover,the more they "test" your game, the less likely they are to lose.You can avoid this problem almost entirely if you write good test plans. But if your tester is also conscious of the difference between playing and testing, he's sure to find bugs beyond the scope of what the average test plan can predict. Occasionally remind your testers that you're paying them to do a job. They can play games on their own time. While they're on the clock,they are there to test.

3. BE FLEXIBLE.

Video games represent a truly collaborative art form. Everyone involved in the game development process needs to work with at least one other person and working with someone else requires flexibility. So, how can Q/A teams be flexible?

Q/A teams are always doing more for their game (and their company) than simply testing the latest build for bugs. This commandment recognizes and encourages that. Be flexible enough to allow your testers to demo their game for the press. Help your marketing and PR teams with screenshots and/or videos. After all, who knows how to avoid all the embarrassing bugs better than the people who found them? The Q/A teamknows the workarounds better than anyone. If your company is flexible enough to allow the Q/A team to help out with this work, the results might be surprising.

Be cautious not to let your Q/A team take on too much work, however. Don't take on so much extra work that you find your team testing less than they should. If your Q/A team ceases to find important bugs, all those screenshots and videos won't amount to much.

Your Q/A team should also be flexible with the bugs they report. As a game nears completion, Q/A will invariably disagree with the production/development team on some of the bugs they want to close out. Be flexible enough to let less important bugs go (all games ship with some bugs, after all), but insist on fixing the bugs that make the biggest difference.

How do you know the difference? If your Q/A team isn't already using some sort of prioritizing scheme in their bug database, get one started. It's as simple as assigning a letter (or number, or both!] to a kind of bug. When it comes time to close out the bugs, it can save you hours upon hours of headaches.

4. FIND AND REPORT BUGS AS EARLY AS POSSIBLE.

This is a tricky one. Q/A is usually employed at the end of the development process, which is absolutely the right way to build a game. This commandment isn't condoning full-blown testing before your alpha milestone. Doing so would only clog the bug database with annoying bugs like "game-has-no-sound."

However, it is helpful to have a lead tester review the save flow( for example) long before it's implemented in the game. If an experienced tester can look at a save flow system design and tell you what will fail a technical requirements checklist, it'll save both the Q/A team and the development team hours or even days of work down the line.

You could also ask a few testers to take a look at the first iterations of your control scheme. Don't tell them how to control the game, just give them the controller and ask them to play around a little. If they can intuitively understand how to interactwith the game, then you know you're on the right track with your controls. If they have problems with certain aspects of the controls, you know what to work on next. This won't give you the feedback you need if you're wondering how intuitive a casual gamer would find your controls, but it's a good place to start.

You can have your lead tester look at the game's text to verify correct usage of naming conventions long before you've implemented the text into the game. These are things they'll test for anyway once the text is in-game, so why not have them look at it before doing difficult implementation?

The save flow, controls, and text check examples are just that, examples. There are many little things an experienced Q/A tester can do for you before the alpha milestone. If you schedule these things out ahead of time, and make sure the tester knows what he/she should not work on, your development team will have more time to work on more important things and the inevitable crunch time will be less ... crunchy.

5. THINK LIKE A HACKER. BE CREATIVE IN FINDING PROBLEMS WITH THE GAME.

It's relatively easy to find a spelling error or a level load bug. The best testers flex their mentality toward the test cases at hand,ranging from a technically skilled hacker to an anti-intuitive four year old. Crash bugs can be found from altering core files and manipulating fifteen different user interface screens (hacker) to smashing your palm on the keyboard and causing a buffer overrun (four year old). And there are plenty of bugs in between.
Hardcore gamers love to bend the game rules as far as possible. Keeping this commandment in mind can help you avoid shipping with those nasty "exploit" bugs found in multiplayer games (both console and PC). But this commandment doesn't just apply to multiplayer. Thinking like a hacker simply means looking for the flaws buried deep in the game, not just on the surface.

Don't only play the game the way you think the average consumer will. Think of ways to play that no one else will. If the main character is supposed to exit a room by using the door, ask your tester to find other ways out of the room. Can the character use the window? Can he jump through the ceiling? Can he walk through a wall? Approaching a seemingly mundane task and finding a creative way through it can result in a lot of surprising bugs.

6. PUT IN AS MUCH EFFORT WITH YOUR REGRESSION TESTING AS YOU DO WITHYOUR INITIAL TESTING [HALO TESTING].

Finding and reporting the initial bug, as well as properly regressing that bug, are two important and essential job responsibilities for every Q/A tester. However, there is a third area of testing that occurs: Halo testing. No, this isn't about the Bungie game. Halo testing is when a tester checks for newly uncovered or added bugs resulting from the fixed bug they are regressing. Successful Halo testing requires the tester to possess Q/A experience (or an innate talent for finding bugs], intimate knowledge of back-end systems, how the product is built or developed, and guidance by the Q/A management team overseeing the title.

Regression testing is more than just reproducing the initially reported issue. When a fix for a bug is checked in there is always a risk associated with that fix. That risk must be evaluated not only by the programmer, designer, or artist but also by the Q/A lead and tester. Ensuring the bug at hand is fixed is only the first step—the tester must then Halo test around that fix looking for new bugs that could have been a result of that fixed bug.

7. DON'T LET Q/A MEMBERS TEST DESIGNS THEY'VE CREATED.

Otherwise known as the conflict of interest commandment, this one can get you in trouble with your ambitious testers. Every tester, at some point, has a brilliant idea. Be very careful when this happens.

It's human nature to be biased against one's own ideas. Therefore, when a tester's suggestion makes it into the game, don't allow her to regress it. By seeing her idea become reality, she cannot effectively Halo test the new feature. She will be lesslikely to observe any bugs resulting from this feature. Get another tester to put the new feature through its paces. This will ensure that the new feature gets the same objective attention as any other.
This commandment does not discourage testers from making the leap to level designer or some other position in the larger organization. Many industry luminaries started their careers in Q/A, and this commandment recognizes that. But it also servesas a reminder to resist allowing your tester's objectivity to be overshadowed by a really good idea.

8. DON'T WRITE SLOPPY BUGS. SPELL AND GRAMMAR CHECK EVERYTHING.

The most effective testers are not those with the highest Gamerscore, but those possessing exceptional written and verbal communication skills. This is because clearly-communicated bugs get fixed faster and better than sloppy, confusing bugs. Moreover, as a group whose fundamental responsibility is to find other people's mistakes, it's simply embarrassing and unprofessional when Q/A's own work is full of errors.

This commandment doesn't insist that every tester hold a Ph.D. in English. Even if your testers lack perfect grammar skills, the least they can do is spell check theirwork. Most bug databases have a spell check function built in. Insist that your testers use it.

9. TEST EVERYTHING YOU CAN REASONABLY TEST.

Don't ignore a feature just because it gives a good first impression. If it's in the game, test it (and test it often) to ensure it works as designed.

For example, upon initially testing the newest gun in a FPS, a tester will notice that it fires correctly, the correct sound plays,it deals the correct amount of damage to the enemy, and it depletes its ammunition according to spec. Oh! And it's really fun to shoot! At this point, your inexperienced tester (having finished the test plan for this gun) may think he's done. He'll move onto the next gun.

But, what if that new gun, when fired at an explosive crate, crashes the game? It seems tedious, but every feature needs to be tested in every reasonable way. That means firing weapons at every interactive target (and most non-interactive targets]. Make sure everything from the smallest decal to the biggest explosion trigger and play correctly, lest you prematurely approve a new feature.

As another example, at some point we've all heard a developer say, "I only fixed X bug. I didn't touch anything else, so don't worry about testing the whole Y feature." An inexperienced tester will take that developer at his word and only test the change as reported. He would miss the bugs that unexpectedly resulted from the change to "X bug."

A good, thorough test plan (one that includes, for example, causing an explosion with every weapon type) can help you avoid this problem, but no test plan can predict every possibility. Your testers need to be aware that they are ultimately responsible for the stability of the game. They need to make sure everything works to spec in every situation. They also need to understand that a "simple change" to a seemingly innocuous feature can have unexpected results. Never settle for spot-checking and don't say you're "done" testing a feature until you're ready to ship it.

10. WORK UNDER THE ASSUMPTION THAT MOST (IF NOT ALL) BUGS CAN BECONSISTENTLY REPRODUCED.

It's only a question of how difficult the bug is to repeat. Even themost "random" bug is repeatable given enough time and effort. Granted, it's not always wise to spend time attempting to consistently repeat an elusive bug, but if you can, it's much easier to fix.

This commandment ties into the first commandment (Scientific Method), but also begs the question: How much time should one spend trying to consistently repeat a bug? You might work under the assumption that every bug is repeatable, but is it advisable to consistently repeat every single bug?

Ask yourself—how important is this bug? If the tester "randomly" crashed the game after beating the first boss, isn't it justified to spend a day attempting to consistently repeat it? On the other hand, if the tester found that the credits scrolled unusually quickly once out of 10 viewings, you probably don't need to research the bug for more than a few minutes.

Consider the severity of the bug when you determine how much time to invest in getting it repeated. You know it can be consistently repeated, but you need to use your judgment to determine if it's worth the effort. If your testers can't get the bug to repeat, list how many times they attempted to against how many times they successfully repeated it (i.e. 1 out of 10 times, the credits scrolled at 5x speed). That way, the development team can address the bug knowing it's not yet consistently repeatable.

A GOLDEN RULE

Summing up these 10 commandments into one all-inclusive statement might look something like this: Test scientifically, creatively, and thoroughly enough to catch all the bugs, but not so obsessively as to jeopardize your ship date.

Whether you follow one golden rule or 10 commandments, Q/A work is as much an art as a science. It's at its best when a diverse team passionately comes together to work on a common goal; to do everything they can to help the development team make the best game possible. Hopefully these commandments can inspire you and your Q/A team to better serve that common goal.

The author acknowledges Rob Thompson,
Q/A manager at Sony OnlineEntertainment for his contributions to this article.
NOVEMBER 2007 GAME DEVELOPER