If turnips were watches, I would wear one by my side.
And if "ifs" and "ands"
Were pots and pans,
There'd be no work for tinkers!
If wishing could make things happen, then even the most destitute people would have everything they wanted.
Saturday, November 24, 2007
If wishes were horses, then beggars would ride
Friday, November 16, 2007
A Production Prespective
by Harvey Smith
Creative Director - Midway Austin
My experience with Q/A over the years has been interesting. I started out as a tester in 1993, knowing nothing about software development. I was an enthusiastic video game player, a writer, and open-and-paper RPG fanatic. This list of commandments would have instantly accelerated my learning by a year.
Now, as a creative director, I crave more polish in games relative to everything else, which usually stems from good development and tech practices. Best practices impact gameplay much more than people often acknowledge. Even in cases where development teams are trying something new and ambitious, a fast frame rate, a solid interface, stability and lots of feedback are the real key to enhancing the player's experience.
I especially like number five from this list, about inspiring testers to play creatively. There's always one person on the test team who thinks like that — someone who covers the basic route that 90 percent of mainstream players will take, then tries five or six creative alternatives. After a while, you develop this as an instinct. "Hmm, this front door triggers a scripted door-opening sequence... I wonder what happens if I break the side window and skip the door."
HEARTS ARE MADE FOR BREAKING
Q/A people know that most games can be broken (or will at least expose something that looks silly) if you try hard enough. Citing alt-path problems, then speculating on their likelihood is very useful; sometimes we take things like that and run with them creatively. We might say "If five percent of testers think to try this, what if we bullet-proof it, then make it more attractive or obvious, so that it occurs to maybe 20 percent of players." Working on the DEUS EX games, which were incredibly free-form, this happened a lot. Even working on an FPS like BlackSite, over the last year we've seen lots of cases where testers approach a combat scenario in a way we didn't expect—routes which might be under-supported and cause frustration. In all cases, having great dialogue with Q/A will make the game better.
THE FINAL PUSH
The very concept of "testing" has evolved a lot to include a bunch of best practices for the genre, such as blind usability tests and post production (as a serious phase of development). The game never gets good as fast as it does in the final months, so it pays to invest in post production testing. Ideally you'll finish the game from beginning to end as soon as you can, then spend a ton of time pounding on it, observing new players stumble through it, and looking for opportunities to pay off dramatic moments. Bringing in round after round of people who have never seen BLACKSITE has taught us invaluable lessons about where players get stuck, run out of ammo, get lost, and have the most or least fun. I wish we had three to six more months of that sort of thing — it would make a tremendous difference in the final quality of the game. I think the difference between great and mediocre publishers is the wisdom to invest in the final period of testing and tuning; the discipline to avoid cheating into this phase, even if a game is late.
Creative Director - Midway Austin
My experience with Q/A over the years has been interesting. I started out as a tester in 1993, knowing nothing about software development. I was an enthusiastic video game player, a writer, and open-and-paper RPG fanatic. This list of commandments would have instantly accelerated my learning by a year.
Now, as a creative director, I crave more polish in games relative to everything else, which usually stems from good development and tech practices. Best practices impact gameplay much more than people often acknowledge. Even in cases where development teams are trying something new and ambitious, a fast frame rate, a solid interface, stability and lots of feedback are the real key to enhancing the player's experience.
I especially like number five from this list, about inspiring testers to play creatively. There's always one person on the test team who thinks like that — someone who covers the basic route that 90 percent of mainstream players will take, then tries five or six creative alternatives. After a while, you develop this as an instinct. "Hmm, this front door triggers a scripted door-opening sequence... I wonder what happens if I break the side window and skip the door."
HEARTS ARE MADE FOR BREAKING
Q/A people know that most games can be broken (or will at least expose something that looks silly) if you try hard enough. Citing alt-path problems, then speculating on their likelihood is very useful; sometimes we take things like that and run with them creatively. We might say "If five percent of testers think to try this, what if we bullet-proof it, then make it more attractive or obvious, so that it occurs to maybe 20 percent of players." Working on the DEUS EX games, which were incredibly free-form, this happened a lot. Even working on an FPS like BlackSite, over the last year we've seen lots of cases where testers approach a combat scenario in a way we didn't expect—routes which might be under-supported and cause frustration. In all cases, having great dialogue with Q/A will make the game better.
THE FINAL PUSH
The very concept of "testing" has evolved a lot to include a bunch of best practices for the genre, such as blind usability tests and post production (as a serious phase of development). The game never gets good as fast as it does in the final months, so it pays to invest in post production testing. Ideally you'll finish the game from beginning to end as soon as you can, then spend a ton of time pounding on it, observing new players stumble through it, and looking for opportunities to pay off dramatic moments. Bringing in round after round of people who have never seen BLACKSITE has taught us invaluable lessons about where players get stuck, run out of ammo, get lost, and have the most or least fun. I wish we had three to six more months of that sort of thing — it would make a tremendous difference in the final quality of the game. I think the difference between great and mediocre publishers is the wisdom to invest in the final period of testing and tuning; the discipline to avoid cheating into this phase, even if a game is late.
Thursday, November 08, 2007
Commandments of Quality Assurance
by Chuck McFadden
1. Employ the Scientific Method.
2. Understand the difference between playing a game and testing a game. Spend most of your time doing the latter.
3. Be flexible.
4. Find and report bugs as early as possible.
5. Think like a hacker. Be creative in finding problems with the game.
6. Put in as much effort with your regression testing as you do with your initial testing.
7. Don't let Q/A members test designs they've created.
8. Don't write sloppy bugs. Spell and grammar check everything.
9. Test everything you can reasonably test.
10. Work under the assumption that most (if not all) bugs can be consistently reproduced.
IF Q/A HAD A SET OF 10 COMMANDMENTS, WHAT WOULD THEY BE?
Approximately two years ago, a member of the IGDA's Q/A Special InterestGroup [SIG] posed that very question, which ignited the SIG into lengthy discussions and friendly arguments until, weeks later, we settled on 10. They were compiled and listed on the SIG site, where they have remained in relative obscurity... until now.
This article lists the commandments in no particular order and examines each in detail. Each has equal value; no one commandment is more important thanany other. But ignore them and the god of game development will smite you.
1. EMPLOY THE SCIENTIFIC METHOD.
When you boil it down, this is what separates a good tester froma bad tester: the good tester—consciously or unconsciously—uses the Scientific Method. Make this a part of your Q/A team's training process, just in case one of your testers paid as little attention during science class as this author. If trained well in the Scientific Method, upon observing a bug, a tester will:
a) Observe and describe. At the least, the tester should take notes on what happened. Ideally, your tester is recording everything (with a VCR or digitally), which helps the tester with the next few steps. Either way, this step simply states what happened, for example that the game crashed at the second save point.
b) Formulate a hypothesis. The tester should speculate on what might have caused the bug. Did they try to save when the game was streaming off the disc? Was a character talking when the save action triggered?
Did the character "die" at the sametime the save initialized? Based on the observed behavior, a tester can develop a testable hypothesis to repeat the bug. (An inexperienced or less-thorough tester will end the process here, writing up the bug based on his/her speculation.)
c) Experiment. The hypothesis should then be tested to see if it results in the initially observed behavior/bug. Re-experiment to narrow down the steps to repeat the bug. If the hypothesis is true, move onto the next step. If the hypothesis is false, refer to step A and start again at step B.
d) Draw a conclusion and communicate the results. Once the tester has verified that they can repeat the bug with the minimal number of steps (in other words, once they have verified that the hypothesis is true), they should write it up. How your tester writes up the bug is up to you and your established procedure. Regardless of how they write up the bug, if they use this method, the bug will be solidly researched and consistently reproducible.
2. UNDERSTAND THE DIFFERENCE BETWEEN PLAYING A GAME AND TESTING A GAME. SPEND MOST OF YOUR TIME DOING THE LATTER.
The most common misconception I've seen with new hires (and with industry colleagues who don't know much about Q/A) is they believe testers "play games all day." If this is 100 percenttrue with your Q/A team, you release terribly buggy games. Quality assurance is not just a matter of testing how a game issupposed to work, but how the game is not supposed to work.
The difference is simple. A good tester knows that losingneeds to be tested as thoroughly as winning. He understandsthat getting the best lap time in a racing game is only half thejob. Otherwise, a bug as simple as a crash at the "game over"screen will never come up if the tester is always playing to win.
This commandment also speaks to another philosophy: Don't exclusively hire "hardcore" gamers. While employing hardcore gamers is important, it's equally important to cultivate more casual players. As a lead or manager, you'll be tempted to hire the most dedicated gamers in your neighborhood. Unfortunately those testers invariably have the hardest time losing. Moreover,the more they "test" your game, the less likely they are to lose.You can avoid this problem almost entirely if you write good test plans. But if your tester is also conscious of the difference between playing and testing, he's sure to find bugs beyond the scope of what the average test plan can predict. Occasionally remind your testers that you're paying them to do a job. They can play games on their own time. While they're on the clock,they are there to test.
3. BE FLEXIBLE.
Video games represent a truly collaborative art form. Everyone involved in the game development process needs to work with at least one other person and working with someone else requires flexibility. So, how can Q/A teams be flexible?
Q/A teams are always doing more for their game (and their company) than simply testing the latest build for bugs. This commandment recognizes and encourages that. Be flexible enough to allow your testers to demo their game for the press. Help your marketing and PR teams with screenshots and/or videos. After all, who knows how to avoid all the embarrassing bugs better than the people who found them? The Q/A teamknows the workarounds better than anyone. If your company is flexible enough to allow the Q/A team to help out with this work, the results might be surprising.
Be cautious not to let your Q/A team take on too much work, however. Don't take on so much extra work that you find your team testing less than they should. If your Q/A team ceases to find important bugs, all those screenshots and videos won't amount to much.
Your Q/A team should also be flexible with the bugs they report. As a game nears completion, Q/A will invariably disagree with the production/development team on some of the bugs they want to close out. Be flexible enough to let less important bugs go (all games ship with some bugs, after all), but insist on fixing the bugs that make the biggest difference.
How do you know the difference? If your Q/A team isn't already using some sort of prioritizing scheme in their bug database, get one started. It's as simple as assigning a letter (or number, or both!] to a kind of bug. When it comes time to close out the bugs, it can save you hours upon hours of headaches.
4. FIND AND REPORT BUGS AS EARLY AS POSSIBLE.
This is a tricky one. Q/A is usually employed at the end of the development process, which is absolutely the right way to build a game. This commandment isn't condoning full-blown testing before your alpha milestone. Doing so would only clog the bug database with annoying bugs like "game-has-no-sound."
However, it is helpful to have a lead tester review the save flow( for example) long before it's implemented in the game. If an experienced tester can look at a save flow system design and tell you what will fail a technical requirements checklist, it'll save both the Q/A team and the development team hours or even days of work down the line.
You could also ask a few testers to take a look at the first iterations of your control scheme. Don't tell them how to control the game, just give them the controller and ask them to play around a little. If they can intuitively understand how to interactwith the game, then you know you're on the right track with your controls. If they have problems with certain aspects of the controls, you know what to work on next. This won't give you the feedback you need if you're wondering how intuitive a casual gamer would find your controls, but it's a good place to start.
You can have your lead tester look at the game's text to verify correct usage of naming conventions long before you've implemented the text into the game. These are things they'll test for anyway once the text is in-game, so why not have them look at it before doing difficult implementation?
The save flow, controls, and text check examples are just that, examples. There are many little things an experienced Q/A tester can do for you before the alpha milestone. If you schedule these things out ahead of time, and make sure the tester knows what he/she should not work on, your development team will have more time to work on more important things and the inevitable crunch time will be less ... crunchy.
5. THINK LIKE A HACKER. BE CREATIVE IN FINDING PROBLEMS WITH THE GAME.
It's relatively easy to find a spelling error or a level load bug. The best testers flex their mentality toward the test cases at hand,ranging from a technically skilled hacker to an anti-intuitive four year old. Crash bugs can be found from altering core files and manipulating fifteen different user interface screens (hacker) to smashing your palm on the keyboard and causing a buffer overrun (four year old). And there are plenty of bugs in between.
Hardcore gamers love to bend the game rules as far as possible. Keeping this commandment in mind can help you avoid shipping with those nasty "exploit" bugs found in multiplayer games (both console and PC). But this commandment doesn't just apply to multiplayer. Thinking like a hacker simply means looking for the flaws buried deep in the game, not just on the surface.
Don't only play the game the way you think the average consumer will. Think of ways to play that no one else will. If the main character is supposed to exit a room by using the door, ask your tester to find other ways out of the room. Can the character use the window? Can he jump through the ceiling? Can he walk through a wall? Approaching a seemingly mundane task and finding a creative way through it can result in a lot of surprising bugs.
6. PUT IN AS MUCH EFFORT WITH YOUR REGRESSION TESTING AS YOU DO WITHYOUR INITIAL TESTING [HALO TESTING].
Finding and reporting the initial bug, as well as properly regressing that bug, are two important and essential job responsibilities for every Q/A tester. However, there is a third area of testing that occurs: Halo testing. No, this isn't about the Bungie game. Halo testing is when a tester checks for newly uncovered or added bugs resulting from the fixed bug they are regressing. Successful Halo testing requires the tester to possess Q/A experience (or an innate talent for finding bugs], intimate knowledge of back-end systems, how the product is built or developed, and guidance by the Q/A management team overseeing the title.
Regression testing is more than just reproducing the initially reported issue. When a fix for a bug is checked in there is always a risk associated with that fix. That risk must be evaluated not only by the programmer, designer, or artist but also by the Q/A lead and tester. Ensuring the bug at hand is fixed is only the first step—the tester must then Halo test around that fix looking for new bugs that could have been a result of that fixed bug.
7. DON'T LET Q/A MEMBERS TEST DESIGNS THEY'VE CREATED.
Otherwise known as the conflict of interest commandment, this one can get you in trouble with your ambitious testers. Every tester, at some point, has a brilliant idea. Be very careful when this happens.
It's human nature to be biased against one's own ideas. Therefore, when a tester's suggestion makes it into the game, don't allow her to regress it. By seeing her idea become reality, she cannot effectively Halo test the new feature. She will be lesslikely to observe any bugs resulting from this feature. Get another tester to put the new feature through its paces. This will ensure that the new feature gets the same objective attention as any other.
This commandment does not discourage testers from making the leap to level designer or some other position in the larger organization. Many industry luminaries started their careers in Q/A, and this commandment recognizes that. But it also servesas a reminder to resist allowing your tester's objectivity to be overshadowed by a really good idea.
8. DON'T WRITE SLOPPY BUGS. SPELL AND GRAMMAR CHECK EVERYTHING.
The most effective testers are not those with the highest Gamerscore, but those possessing exceptional written and verbal communication skills. This is because clearly-communicated bugs get fixed faster and better than sloppy, confusing bugs. Moreover, as a group whose fundamental responsibility is to find other people's mistakes, it's simply embarrassing and unprofessional when Q/A's own work is full of errors.
This commandment doesn't insist that every tester hold a Ph.D. in English. Even if your testers lack perfect grammar skills, the least they can do is spell check theirwork. Most bug databases have a spell check function built in. Insist that your testers use it.
9. TEST EVERYTHING YOU CAN REASONABLY TEST.
Don't ignore a feature just because it gives a good first impression. If it's in the game, test it (and test it often) to ensure it works as designed.
For example, upon initially testing the newest gun in a FPS, a tester will notice that it fires correctly, the correct sound plays,it deals the correct amount of damage to the enemy, and it depletes its ammunition according to spec. Oh! And it's really fun to shoot! At this point, your inexperienced tester (having finished the test plan for this gun) may think he's done. He'll move onto the next gun.
But, what if that new gun, when fired at an explosive crate, crashes the game? It seems tedious, but every feature needs to be tested in every reasonable way. That means firing weapons at every interactive target (and most non-interactive targets]. Make sure everything from the smallest decal to the biggest explosion trigger and play correctly, lest you prematurely approve a new feature.
As another example, at some point we've all heard a developer say, "I only fixed X bug. I didn't touch anything else, so don't worry about testing the whole Y feature." An inexperienced tester will take that developer at his word and only test the change as reported. He would miss the bugs that unexpectedly resulted from the change to "X bug."
A good, thorough test plan (one that includes, for example, causing an explosion with every weapon type) can help you avoid this problem, but no test plan can predict every possibility. Your testers need to be aware that they are ultimately responsible for the stability of the game. They need to make sure everything works to spec in every situation. They also need to understand that a "simple change" to a seemingly innocuous feature can have unexpected results. Never settle for spot-checking and don't say you're "done" testing a feature until you're ready to ship it.
10. WORK UNDER THE ASSUMPTION THAT MOST (IF NOT ALL) BUGS CAN BECONSISTENTLY REPRODUCED.
It's only a question of how difficult the bug is to repeat. Even themost "random" bug is repeatable given enough time and effort. Granted, it's not always wise to spend time attempting to consistently repeat an elusive bug, but if you can, it's much easier to fix.
This commandment ties into the first commandment (Scientific Method), but also begs the question: How much time should one spend trying to consistently repeat a bug? You might work under the assumption that every bug is repeatable, but is it advisable to consistently repeat every single bug?
Ask yourself—how important is this bug? If the tester "randomly" crashed the game after beating the first boss, isn't it justified to spend a day attempting to consistently repeat it? On the other hand, if the tester found that the credits scrolled unusually quickly once out of 10 viewings, you probably don't need to research the bug for more than a few minutes.
Consider the severity of the bug when you determine how much time to invest in getting it repeated. You know it can be consistently repeated, but you need to use your judgment to determine if it's worth the effort. If your testers can't get the bug to repeat, list how many times they attempted to against how many times they successfully repeated it (i.e. 1 out of 10 times, the credits scrolled at 5x speed). That way, the development team can address the bug knowing it's not yet consistently repeatable.
A GOLDEN RULE
Summing up these 10 commandments into one all-inclusive statement might look something like this: Test scientifically, creatively, and thoroughly enough to catch all the bugs, but not so obsessively as to jeopardize your ship date.
Whether you follow one golden rule or 10 commandments, Q/A work is as much an art as a science. It's at its best when a diverse team passionately comes together to work on a common goal; to do everything they can to help the development team make the best game possible. Hopefully these commandments can inspire you and your Q/A team to better serve that common goal.
The author acknowledges Rob Thompson,
Q/A manager at Sony OnlineEntertainment for his contributions to this article.
NOVEMBER 2007 GAME DEVELOPER
1. Employ the Scientific Method.
2. Understand the difference between playing a game and testing a game. Spend most of your time doing the latter.
3. Be flexible.
4. Find and report bugs as early as possible.
5. Think like a hacker. Be creative in finding problems with the game.
6. Put in as much effort with your regression testing as you do with your initial testing.
7. Don't let Q/A members test designs they've created.
8. Don't write sloppy bugs. Spell and grammar check everything.
9. Test everything you can reasonably test.
10. Work under the assumption that most (if not all) bugs can be consistently reproduced.
IF Q/A HAD A SET OF 10 COMMANDMENTS, WHAT WOULD THEY BE?
Approximately two years ago, a member of the IGDA's Q/A Special InterestGroup [SIG] posed that very question, which ignited the SIG into lengthy discussions and friendly arguments until, weeks later, we settled on 10. They were compiled and listed on the SIG site, where they have remained in relative obscurity... until now.
This article lists the commandments in no particular order and examines each in detail. Each has equal value; no one commandment is more important thanany other. But ignore them and the god of game development will smite you.
1. EMPLOY THE SCIENTIFIC METHOD.
When you boil it down, this is what separates a good tester froma bad tester: the good tester—consciously or unconsciously—uses the Scientific Method. Make this a part of your Q/A team's training process, just in case one of your testers paid as little attention during science class as this author. If trained well in the Scientific Method, upon observing a bug, a tester will:
a) Observe and describe. At the least, the tester should take notes on what happened. Ideally, your tester is recording everything (with a VCR or digitally), which helps the tester with the next few steps. Either way, this step simply states what happened, for example that the game crashed at the second save point.
b) Formulate a hypothesis. The tester should speculate on what might have caused the bug. Did they try to save when the game was streaming off the disc? Was a character talking when the save action triggered?
Did the character "die" at the sametime the save initialized? Based on the observed behavior, a tester can develop a testable hypothesis to repeat the bug. (An inexperienced or less-thorough tester will end the process here, writing up the bug based on his/her speculation.)
c) Experiment. The hypothesis should then be tested to see if it results in the initially observed behavior/bug. Re-experiment to narrow down the steps to repeat the bug. If the hypothesis is true, move onto the next step. If the hypothesis is false, refer to step A and start again at step B.
d) Draw a conclusion and communicate the results. Once the tester has verified that they can repeat the bug with the minimal number of steps (in other words, once they have verified that the hypothesis is true), they should write it up. How your tester writes up the bug is up to you and your established procedure. Regardless of how they write up the bug, if they use this method, the bug will be solidly researched and consistently reproducible.
2. UNDERSTAND THE DIFFERENCE BETWEEN PLAYING A GAME AND TESTING A GAME. SPEND MOST OF YOUR TIME DOING THE LATTER.
The most common misconception I've seen with new hires (and with industry colleagues who don't know much about Q/A) is they believe testers "play games all day." If this is 100 percenttrue with your Q/A team, you release terribly buggy games. Quality assurance is not just a matter of testing how a game issupposed to work, but how the game is not supposed to work.
The difference is simple. A good tester knows that losingneeds to be tested as thoroughly as winning. He understandsthat getting the best lap time in a racing game is only half thejob. Otherwise, a bug as simple as a crash at the "game over"screen will never come up if the tester is always playing to win.
This commandment also speaks to another philosophy: Don't exclusively hire "hardcore" gamers. While employing hardcore gamers is important, it's equally important to cultivate more casual players. As a lead or manager, you'll be tempted to hire the most dedicated gamers in your neighborhood. Unfortunately those testers invariably have the hardest time losing. Moreover,the more they "test" your game, the less likely they are to lose.You can avoid this problem almost entirely if you write good test plans. But if your tester is also conscious of the difference between playing and testing, he's sure to find bugs beyond the scope of what the average test plan can predict. Occasionally remind your testers that you're paying them to do a job. They can play games on their own time. While they're on the clock,they are there to test.
3. BE FLEXIBLE.
Video games represent a truly collaborative art form. Everyone involved in the game development process needs to work with at least one other person and working with someone else requires flexibility. So, how can Q/A teams be flexible?
Q/A teams are always doing more for their game (and their company) than simply testing the latest build for bugs. This commandment recognizes and encourages that. Be flexible enough to allow your testers to demo their game for the press. Help your marketing and PR teams with screenshots and/or videos. After all, who knows how to avoid all the embarrassing bugs better than the people who found them? The Q/A teamknows the workarounds better than anyone. If your company is flexible enough to allow the Q/A team to help out with this work, the results might be surprising.
Be cautious not to let your Q/A team take on too much work, however. Don't take on so much extra work that you find your team testing less than they should. If your Q/A team ceases to find important bugs, all those screenshots and videos won't amount to much.
Your Q/A team should also be flexible with the bugs they report. As a game nears completion, Q/A will invariably disagree with the production/development team on some of the bugs they want to close out. Be flexible enough to let less important bugs go (all games ship with some bugs, after all), but insist on fixing the bugs that make the biggest difference.
How do you know the difference? If your Q/A team isn't already using some sort of prioritizing scheme in their bug database, get one started. It's as simple as assigning a letter (or number, or both!] to a kind of bug. When it comes time to close out the bugs, it can save you hours upon hours of headaches.
4. FIND AND REPORT BUGS AS EARLY AS POSSIBLE.
This is a tricky one. Q/A is usually employed at the end of the development process, which is absolutely the right way to build a game. This commandment isn't condoning full-blown testing before your alpha milestone. Doing so would only clog the bug database with annoying bugs like "game-has-no-sound."
However, it is helpful to have a lead tester review the save flow( for example) long before it's implemented in the game. If an experienced tester can look at a save flow system design and tell you what will fail a technical requirements checklist, it'll save both the Q/A team and the development team hours or even days of work down the line.
You could also ask a few testers to take a look at the first iterations of your control scheme. Don't tell them how to control the game, just give them the controller and ask them to play around a little. If they can intuitively understand how to interactwith the game, then you know you're on the right track with your controls. If they have problems with certain aspects of the controls, you know what to work on next. This won't give you the feedback you need if you're wondering how intuitive a casual gamer would find your controls, but it's a good place to start.
You can have your lead tester look at the game's text to verify correct usage of naming conventions long before you've implemented the text into the game. These are things they'll test for anyway once the text is in-game, so why not have them look at it before doing difficult implementation?
The save flow, controls, and text check examples are just that, examples. There are many little things an experienced Q/A tester can do for you before the alpha milestone. If you schedule these things out ahead of time, and make sure the tester knows what he/she should not work on, your development team will have more time to work on more important things and the inevitable crunch time will be less ... crunchy.
5. THINK LIKE A HACKER. BE CREATIVE IN FINDING PROBLEMS WITH THE GAME.
It's relatively easy to find a spelling error or a level load bug. The best testers flex their mentality toward the test cases at hand,ranging from a technically skilled hacker to an anti-intuitive four year old. Crash bugs can be found from altering core files and manipulating fifteen different user interface screens (hacker) to smashing your palm on the keyboard and causing a buffer overrun (four year old). And there are plenty of bugs in between.
Hardcore gamers love to bend the game rules as far as possible. Keeping this commandment in mind can help you avoid shipping with those nasty "exploit" bugs found in multiplayer games (both console and PC). But this commandment doesn't just apply to multiplayer. Thinking like a hacker simply means looking for the flaws buried deep in the game, not just on the surface.
Don't only play the game the way you think the average consumer will. Think of ways to play that no one else will. If the main character is supposed to exit a room by using the door, ask your tester to find other ways out of the room. Can the character use the window? Can he jump through the ceiling? Can he walk through a wall? Approaching a seemingly mundane task and finding a creative way through it can result in a lot of surprising bugs.
6. PUT IN AS MUCH EFFORT WITH YOUR REGRESSION TESTING AS YOU DO WITHYOUR INITIAL TESTING [HALO TESTING].
Finding and reporting the initial bug, as well as properly regressing that bug, are two important and essential job responsibilities for every Q/A tester. However, there is a third area of testing that occurs: Halo testing. No, this isn't about the Bungie game. Halo testing is when a tester checks for newly uncovered or added bugs resulting from the fixed bug they are regressing. Successful Halo testing requires the tester to possess Q/A experience (or an innate talent for finding bugs], intimate knowledge of back-end systems, how the product is built or developed, and guidance by the Q/A management team overseeing the title.
Regression testing is more than just reproducing the initially reported issue. When a fix for a bug is checked in there is always a risk associated with that fix. That risk must be evaluated not only by the programmer, designer, or artist but also by the Q/A lead and tester. Ensuring the bug at hand is fixed is only the first step—the tester must then Halo test around that fix looking for new bugs that could have been a result of that fixed bug.
7. DON'T LET Q/A MEMBERS TEST DESIGNS THEY'VE CREATED.
Otherwise known as the conflict of interest commandment, this one can get you in trouble with your ambitious testers. Every tester, at some point, has a brilliant idea. Be very careful when this happens.
It's human nature to be biased against one's own ideas. Therefore, when a tester's suggestion makes it into the game, don't allow her to regress it. By seeing her idea become reality, she cannot effectively Halo test the new feature. She will be lesslikely to observe any bugs resulting from this feature. Get another tester to put the new feature through its paces. This will ensure that the new feature gets the same objective attention as any other.
This commandment does not discourage testers from making the leap to level designer or some other position in the larger organization. Many industry luminaries started their careers in Q/A, and this commandment recognizes that. But it also servesas a reminder to resist allowing your tester's objectivity to be overshadowed by a really good idea.
8. DON'T WRITE SLOPPY BUGS. SPELL AND GRAMMAR CHECK EVERYTHING.
The most effective testers are not those with the highest Gamerscore, but those possessing exceptional written and verbal communication skills. This is because clearly-communicated bugs get fixed faster and better than sloppy, confusing bugs. Moreover, as a group whose fundamental responsibility is to find other people's mistakes, it's simply embarrassing and unprofessional when Q/A's own work is full of errors.
This commandment doesn't insist that every tester hold a Ph.D. in English. Even if your testers lack perfect grammar skills, the least they can do is spell check theirwork. Most bug databases have a spell check function built in. Insist that your testers use it.
9. TEST EVERYTHING YOU CAN REASONABLY TEST.
Don't ignore a feature just because it gives a good first impression. If it's in the game, test it (and test it often) to ensure it works as designed.
For example, upon initially testing the newest gun in a FPS, a tester will notice that it fires correctly, the correct sound plays,it deals the correct amount of damage to the enemy, and it depletes its ammunition according to spec. Oh! And it's really fun to shoot! At this point, your inexperienced tester (having finished the test plan for this gun) may think he's done. He'll move onto the next gun.
But, what if that new gun, when fired at an explosive crate, crashes the game? It seems tedious, but every feature needs to be tested in every reasonable way. That means firing weapons at every interactive target (and most non-interactive targets]. Make sure everything from the smallest decal to the biggest explosion trigger and play correctly, lest you prematurely approve a new feature.
As another example, at some point we've all heard a developer say, "I only fixed X bug. I didn't touch anything else, so don't worry about testing the whole Y feature." An inexperienced tester will take that developer at his word and only test the change as reported. He would miss the bugs that unexpectedly resulted from the change to "X bug."
A good, thorough test plan (one that includes, for example, causing an explosion with every weapon type) can help you avoid this problem, but no test plan can predict every possibility. Your testers need to be aware that they are ultimately responsible for the stability of the game. They need to make sure everything works to spec in every situation. They also need to understand that a "simple change" to a seemingly innocuous feature can have unexpected results. Never settle for spot-checking and don't say you're "done" testing a feature until you're ready to ship it.
10. WORK UNDER THE ASSUMPTION THAT MOST (IF NOT ALL) BUGS CAN BECONSISTENTLY REPRODUCED.
It's only a question of how difficult the bug is to repeat. Even themost "random" bug is repeatable given enough time and effort. Granted, it's not always wise to spend time attempting to consistently repeat an elusive bug, but if you can, it's much easier to fix.
This commandment ties into the first commandment (Scientific Method), but also begs the question: How much time should one spend trying to consistently repeat a bug? You might work under the assumption that every bug is repeatable, but is it advisable to consistently repeat every single bug?
Ask yourself—how important is this bug? If the tester "randomly" crashed the game after beating the first boss, isn't it justified to spend a day attempting to consistently repeat it? On the other hand, if the tester found that the credits scrolled unusually quickly once out of 10 viewings, you probably don't need to research the bug for more than a few minutes.
Consider the severity of the bug when you determine how much time to invest in getting it repeated. You know it can be consistently repeated, but you need to use your judgment to determine if it's worth the effort. If your testers can't get the bug to repeat, list how many times they attempted to against how many times they successfully repeated it (i.e. 1 out of 10 times, the credits scrolled at 5x speed). That way, the development team can address the bug knowing it's not yet consistently repeatable.
A GOLDEN RULE
Summing up these 10 commandments into one all-inclusive statement might look something like this: Test scientifically, creatively, and thoroughly enough to catch all the bugs, but not so obsessively as to jeopardize your ship date.
Whether you follow one golden rule or 10 commandments, Q/A work is as much an art as a science. It's at its best when a diverse team passionately comes together to work on a common goal; to do everything they can to help the development team make the best game possible. Hopefully these commandments can inspire you and your Q/A team to better serve that common goal.
The author acknowledges Rob Thompson,
Q/A manager at Sony OnlineEntertainment for his contributions to this article.
NOVEMBER 2007 GAME DEVELOPER
Friday, November 02, 2007
If in case you don't know the reasons why US troops can't leave Iraq
Here is my list:
1. Maintain the flow of oil and gas in the region.
2. Prevent the estabishment of an Al Qaeda safe haven.
3. Contain Iranian influence.
4. Prevent a regional war.
5. Maintain American military credibility in the region.
6. Prevent a humanitarian catastophe on the scale of Rwanda
1. Maintain the flow of oil and gas in the region.
2. Prevent the estabishment of an Al Qaeda safe haven.
3. Contain Iranian influence.
4. Prevent a regional war.
5. Maintain American military credibility in the region.
6. Prevent a humanitarian catastophe on the scale of Rwanda
Subscribe to:
Posts (Atom)