need tips on testing the game

[wulfsaga]

seriously, the more longer i created game the more time i need to test it!
i mean what if i create 1 hour game, did it means i must test it for one hour too?
I need a tips to make testing more quick and simple (and not start from beginning of game.)
And the biggest problem is when using save, the game will save BEFORE any change on the script. So when it load, it still using script where nothing is fixed.

any tips? @_@

[chensterrain]

Instead of saving, if you use labels at different points in the game and jump to them when you want to test those sections, testing should go a lot faster... (sorry if you're already doing this! I made the mistake of testing using my game saves when I first started, and kept wondering why I kept getting so many random errors... )

e.g.

Code: Select all

label start:
   jump halfwaythroughthegame

***

label halfwaythroughthegame:
   stuff you want to test

[EwanG]

Below is what works for me. YMMV...

Begin with the Lint utility - it will catch the "stupid" things we all do during development.

Next, I go through the script and find all my menus, and then build a walkthrough where I list all the menus, and whether that "path" uses each menu, and if so what choice(s) affect it. That way I can check off a path as it's completed, and know that any fixes shouldn't affect it.

Then I go through the path, and note any problems as I go through (avoiding the temptation to stop and fix as I find, which for me is HARD). Then fix everything, and run again.

Finally after all fixes, I package for distribution, and then I run the longest path (whichever it is) one more time to make sure that packaging didn't introduce a problem, and that there isn't something that missed my eye the first time.

I know you said you wanted to find a way to avoid having a 1 hour game take an hour to test. I don't think you can do a thorough job without spending as long (or longer!) to test as the game is going to take to play.

At work the rule of thumb is 30% requirements, 30% coding, 30% testing. So...

[Jake]

At work the rule of thumb is 30% requirements, 30% coding, 30% testing. So...

Is the other 10% for Procrastination?


The last two MAB titles we've tested by the method of "letting Jasper play them". That said, whenever I've done any significant amount of writing in a VN I've wanted to run through that scene to make sure it was paced the way I thought it was in the editor, and the position of characters was right, and the graphics I'd used had the right names and so on... and just to give me a feeling of satisfaction that I'd got something completed. So I tend to find that a lot of my 'testing' gets done as I work, and it's mostly complex variable interaction which gets left to be picked up by more formal testing.

[IceD]

Those methods are quite good. You might also want to try one of the best - everytime you'd finish a small part of the novel, try to test it - developer tools inside Ren'Py are a good help. It's better to test alwyas a bit by bit, rather than making mistake somewhere halfway and after finishing the game rack your brain where the darn error is (which is very annoying).

When you finish working on the project, there's always a good way to run a small beta test, letting someone to check the game, for example - close friends or other people you can trust enough to do so. Letting someone test the game is the best way to fight with bugs ad errors; people who weren't working on the title have a different view upon things, and tend to spot certain bugs more likely than developers, who look onto their work from a different, more "mechanical" perspective.

The last two MAB titles we've tested by the method of "letting Jasper play them".

Whoa, thats an abuse!

[chunderbunny]

The last two MAB titles we've tested by the method of "letting Jasper play them".

It turns out I'm really good at breaking stuff.

Seriously though, testing stuff is hard to do right. It is often impractical to play through every combination of choices (even if you speed-read, like me) so you end up testing short sections, then a couple of longer sections and finally as many complete paths as you can stomach.

If you have a complex system of variables it is probably a good idea to make a tree of all possible actions which can affect the variables and see if you can spot any logical flaws or inconsistencies. Try and test all the extreme cases (they are the ones most likely to break IME) such as "getting every question wrong" or "always choosing heads". This will hopefully catch any divide-by-zero errors (assuming you are doing any division) which might otherwise go unnoticed.

[ficedula]

If you have a complex system of variables it is probably a good idea to make a tree of all possible actions which can affect the variables and see if you can spot any logical flaws or inconsistencies. Try and test all the extreme cases (they are the ones most likely to break IME) such as "getting every question wrong" or "always choosing heads". This will hopefully catch any divide-by-zero errors (assuming you are doing any division) which might otherwise go unnoticed.

If you have a non-linear game (see also: MorningStar) then building a tree of all possible actions is probably "an intractable problem", so it becomes even more important to basically try and break things by picking odd non-obvious combinations and extreme cases. For example (tedious as it might be!) if you have a choice of subjects to study, what happens if you study *nothing* but one subject all game? Can you get that subject above 100%, or the other subjects to drop below 0%? Admittedly going through the whole game like that could be tedious unless you have programmer assistance.

[Wintermoon]

It helps to have an understanding of the system that you are testing. It also helps to have debug tools that let you examine the value of variables that are usually hidden from the player.

If two choices are truly independent, then you don't need to test combinations. If several choices affect the same variable, test the extremes (high and low) and maybe a couple in between.

Any time the game branches, due to direct choice or a variable, test that branch point. If a choice is supposed to change a variable, test to make sure the variable really changes. If the game tests has a good ending for more than 90 love points, a bad ending for less than 10 love points, and a normal ending for everything in between, test those three possibilities but don't bother testing the difference between 20 and 80 love points if they lead to the same ending.

Finally, play through a couple of times with different goals. Try to pursue any ending. Try to go on as many dates as possible. Try to get all the characters to hate you. Be anti-social. Be a pathological liar. Be completely random. Try to create contradictory conditions. Check how the game responds and if the response makes sense.

Whenever you fix any errors, you need to restart testing from the beginning. Anything that you already tested has to be tested again to see if it broke. Therefore, it makes sense to try to fix as many errors as possible in each iteration, so you might want to keep testing the old version for some time before starting over again with the new version.

Overall, testing should take at least as long as development. Be as thorough as possible. Then expect to receive a long list of new bug reports when you release the game, because no matter how thorough you are, bugs always slip through.

[Jake]

It helps to have an understanding of the system that you are testing.

This is a double-edged sword; having an understanding of the system can also lead you to make the same kind of assumptions about how it works or should work which the programmer badly made when he created the bugs in the first place; testing with inside-knowledge and debug variable access makes you more likely to test that the code does what the programmer wrote rather than what the programmer intended. The user is never going to see these debug variables, so it matters less whether the var has the right value and more whether the observable behaviour is correct.

In an ideal world you would have two sets of testers; some who understand how the system works and can test that the behind-the-scenes systems are functioning as expected, and others who can perform black-box testing and can tell you that the system is behaving correctly from a user's point of view.

[EwanG]

In an ideal world you would have two sets of testers;

In an ideal world there would be automated test tools that were affordable to run against our games so that even if it takes several hours, the author doesn't have to sit around and do it

While I have worked with some decent scriptable test tools, I'm not aware of any good free ones. By good, I mean take less time to setup than running the tests myself...

[Jake]

While I have worked with some decent scriptable test tools, I'm not aware of any good free ones. By good, I mean take less time to setup than running the tests myself...

We use both scriptable testing and unit testing tools where I work; the benefit isn't so much for testing the first time, but for testing after you've made changes to make sure the rest of the system still works the way you expected, and for testing huge amounts of concurrent activity which would be impractical to test manually.

Of course, neither of these are really things that VN testers can benefit from, in most cases.

[Wintermoon]

This is a double-edged sword; having an understanding of the system can also lead you to make the same kind of assumptions about how it works or should work which the programmer badly made when he created the bugs in the first place; testing with inside-knowledge and debug variable access makes you more likely to test that the code does what the programmer wrote rather than what the programmer intended. The user is never going to see these debug variables, so it matters less whether the var has the right value and more whether the observable behaviour is correct.

White box testing is not a replacement for black box testing, but it is an important tool that should not be discarded lightly. Take, for example, the stereotypical "love point" scenario. Black box testing can reveal that you can one girl's good ending by mostly being nice to that girl, but the only way to test if being nice in any specific case actually increases love points without inspecting the love points variable is to play through the game twice for every choice, keeping all choices the same except for the one being tested, such that choosing one option leads to one ending and choosing the other option leads to another ending. Not exactly reasonable without automated tools.

In an ideal world there would be automated test tools that were affordable to run against our games so that even if it takes several hours, the author doesn't have to sit around and do it

It shouldn't be that hard to create a Ren'Py-specific tool that quickly runs through the game making a specific set of choices and confirms that a specific ending is reached without crashing. However, such a tool could not catch grammar errors, spelling errors, plot inconsistencies, incorrect sprites, incorrect backgrounds, incorrectly attributed speech, incorrect facial expressions, choices with illogical outcomes, or any number of other soft errors, and it would fall apart as soon as any random factor is introduced into the game.

[Midnighticequeen]

It shouldn't be that hard to create a Ren'Py-specific tool that quickly runs through the game making a specific set of choices and confirms that a specific ending is reached without crashing. However, such a tool could not catch grammar errors, spelling errors, plot inconsistencies, incorrect sprites, incorrect backgrounds, incorrectly attributed speech, incorrect facial expressions, choices with illogical outcomes, or any number of other soft errors, and it would fall apart as soon as any random factor is introduced into the game.

Wow! I road to testing a game is a long one and it can seem quite tedious, but it's well worth the effort in the long run. Also, having a lot of friends help you by beta testing wouldn't hurt.