In Harmony QA Plan Documentation

Now that In Harmony is complete, I want to share some of our documentation that we used to plan and schedule the Alpha phase, which ended in us successfully deploying the game for public download on the Xbox One platform. Some of these thoughts may be rough around the edges, because they were made for internal use and presentation.

Below is our QA plan generated last July, as we entered Alpha phase.

QA Plan OR How I Learned to stop worrying and love the bugs

Alpha – Content complete. All features should be implemented. They may need to be tweaked, reworked, or improved, and we can work on that during this phase, but nothing new should be entering the game, functionally.

Bug Plan – We have the apparatus already set up, but it will need to be improved and expanded upon during the Alpha phase. We need to have a bug tracking and planning meeting every week, replacing the sprint planning for programmers. Details on this below.

• Reporting – Anyone on the team can report a bug using the Google Form, and it gets added to the queue in Trello and everyone is notified on Slack. This is ideal, and we’ll keep this system in place going forward. The only thing that needs to be updated is it needs to be more obvious who reported the bug.

  • Unique Identifier – It’s important to keep these descriptive and unique, or things will get confusing real fast.

  • Reporter – Imperative to ensure any questions are directed to the individual able to help

  • Severity – We’re doing A, B, C, and D, just with different names to make it easier to work with. (Showstopper, High, Medium, Low)

  • Environment – Build, Engine Play Mode, Engine Editing

  • Steps to reproduce – Imperative for programmer solving the bug o Who should be notified – Useful for assignment and prioritization

  • Unreal crash report – Useful for identifying sources of crashes

  • Images/Videos – Optional attachments

• Testing – This is a great place to allocate designer time: A simple task to have designers play the game for 6 hours a week during the Alpha phase and report all the bugs they experience.

  • Play in Build - We’ll be trying to resolve bugs on our target platform. Weekly builds are pushed to Xbox. To solve the issue of Xboxes only being available on campus, 25% of the testing period will be on Xbox, with the rest on PC build.

  • Different severities – Most of our reported bugs so far are showstoppers. This is because they are the easiest to notice. However, we need to make sure it becomes a more even balance, so the programmers aren’t just putting out extreme fires all the time, and we’re actively improving the game as we go. We won’t have bug quotas per designer, but it will be encouraged to be thorough QA testers.

• Allocation and Tracking – So far tracking has been on point, but allocation needs to be officiated in the Alpha phase. Based on our bug counts, hours and bugs need to be allocated to each programmer weekly.

  • Reproduction – The first step of allocating a bug needs to be reproducing it. Reporter needs to be on hand to clarify their reproduction instructions and help the programmer. If a bug cannot be reproduced, they need to tackle a different bug.

  • Severity – Need to reprioritize the bug count weekly and give people bugs spread across the spectrum of severity. The importance of this is threefold. One, the programmers need to break up big crises with easier stuff to stay sane. Second, we need to fix all showstopper bugs, A bugs are almost never known shippable. If we solely focus on A bugs, we’ll never fix any of the other bugs and the game will ship with them. Gross. Third, we need to be able to track who’s good at solving which kind of bug. If someone is particularly fast and efficient in one area, or Charles is really good at finding and fixing Ryan’s reported bugs, we can keep them together.

  • Dupe tracking – With so many people playing and reporting on the game, it’s inevitable some bugs will be confusing or duplicates. This is where it’ll be important to make sure everyone has the right perforce revision and duplicate bugs are archived. This will be part of the Bug Planning Weekly meeting that we will have.

• Validation

  • Kick it back: The designer who reported the bug needs to be the one testing if it’s fixed. A programmer will send a bug to needs evaluation, then ping the reporter. The reporter will have 24 hours to test the bug and mark it fully fixed and validated.

  • Perforce revision: Added a new field to the survey, that specifies the Perforce revision number the bug was seen in. Once the bug is fixed, the revision number will be added to the Trello card. This will make dupe tracking and build control. easier.

Build control – Need to be disciplined about the build. We will have a weekly build with all known fixed bugs listed and the build locked. These will be bugs that have gone through the full workflow including validation, and have the proper Perforce revision number marked on their Trello ticket.

Risk Plan – Need to identify the greatest tech risks. What will get in our way of getting the game working on Xbox, and how will we deal with them? Examples that come to mind:

• Xbox Optimization

• Lighting Iteration

• Certification

Release/Promotion Plan

• Websites – I already have the website: inharmonygame.com. It’s a basic website right now, made in Google sites, but the domain can support wordpress or squarespace. We will allocate designer time for this during Alpha phase.

• Social media – I have the accounts set up, we just need to schedule out posts and start promoting the game. Ryan would be ideal for this, put him on it once we are Alpha locked.

• Ad graphics – Posters, logos, screenshots, everything else. Gabby says she would love to help us with this, and we could use the upstairs artists and designers for everything else.

• Xbox store requirements – Consult the Xbox cheat sheet for this, there’s a lot of crossover with Ad graphics.

• Awards – Intel, The Game Awards, IGF

Beta Date – The date where bugs equal 0 and we’re ready to make our first submission to Microsoft.

• Crossover date – We have 45 total bugs right now, and we’re averaging 9 bugs reported per week, and for the past two weeks we have 21 fixed bugs as of right now (which has been spread out over a few weeks). Factoring in a 30% increase in bug reports during Alpha and a proposed crossover date of 7/17, we need an average bug fix rate of 19 bugs per week for the next 3 weeks, or 4 bugs per programmer + Project Lead.

• Iterations – By targeting 7/17 that gives us 2.5 weeks of float if we’re behind on fixing showstoppers or we fail cert. Cert only takes 24 hours on average, but we should be maintaining build discipline as mentioned above, meaning that we should only push a verified build at the end of a week. That gives us two iteration cycles before final submission.