Lemma Soft Forums

I have a whole lot of questions about interacting with testers and running closed tests since I'm gunning for an alpha build for a personal project sometime in June. If anyone with experience running closed alpha/beta testing OR anyone who has done closed alpha/beta testing for VNs could answer any of them I'd be really grateful.

Distribution:
How do you distribute your game to your testers?

• Do you send the game folders and they open and test from the Renpy launcher?
  Do you build distributions and make those available for DL?

If you distribute builds, do you archive any of the files or keep them open?

Feedback:
How do you gather feedback from your testers?

• Do you ask them to answer specific questions or just ask for general impressions/feedback?
  Do you provide a form they can fill out or just send them a list of questions?

What methods of gathering feedback seems to work best? Worst?

For Devs and Testers:
What is a reasonable amount of time for an alpha/beta testing period to run if the testers are volunteers?
How many versions per testing period is reasonable?

If you have run an alpha/beta test:
What are some things to avoid when asking for people to test for you (esp. volunteers)?
How do you avoid "bad" testers? (e.g. People who say they want to test but don't deliver feedback)
Do you split up testing into storylines/paths so that Test Group A only plays Anna's route and Test Group B only plays Lily's route?
For longer games, how much do you expect testers to test? Only one route? The whole thing? As much as they can?

For people who test VNs/are interested in testing VNs:
What do you look for in an alpha/beta VN that makes you want to test it as a volunteer?
Are you a completionist? If not, would you be willing to play until 100% completion for a testing period?
How do you feel about testing a certain route vs being free to test the whole thing?
Would you prefer a feedback form to fill out or just to tell the developer your general feelings/critique of the game?
What mode of communication do you feel is best for you to give feedback to the dev? (e.g. PM, e-mail, form, live chat, etc)
How many builds would you be willing to test as a volunteer? (e.g. only alpha versions, only beta versions, both, just once, etc)
How much time would you be willing to spend testing a game? (hours? days? until completion/release?)

As you've probably noticed, I'm mostly wondering about dealing with volunteer testers. Would love to hear people's opinions/answers.

I'll touch on a few points from my limited experience.

Distribute however you like.
It's up to you if you want testers to be able to view files or not. If you are unsure, archive them first.

It's good to have different people as testers. People with a lot of experience with the genre, people who are kind of familiar, and some who generally have no idea what they are doing. You'll get different kinds of feedback from each group, all of it valuable.

It couldn't hurt to give them criteria for feedback, but also allow them to say what they like. Make sure to let them know how direct you want them to be. Be honest, so you don't end up getting your feeling hurt.

Personally I've added a feedback feature to my game (just for test versions), so the testers can click the Generate Feedback button, and it'll create a lot of useful information on how they played, and save it to a text file that they can email to me. That way I have a lot of detailed information that the testers don't need to write, leaving them free to write about other things.

The kind of information in the text file includes choices they've made, how long they took to read text at different stages of the game (to see where they got bored), how long they took to make decisions, etc. It's written in natural language so the testers can see what kind of information I'll be able to see, and to make it easier for me to read.

TrickWithAKnife wrote:Personally I've added a feedback feature to my game (just for test versions), so the testers can click the Generate Feedback button, and it'll create a lot of useful information on how they played, and save it to a text file that they can email to me. That way I have a lot of detailed information that the testers don't need to write, leaving them free to write about other things.

The kind of information in the text file includes choices they've made, how long they took to read text at different stages of the game (to see where they got bored), how long they took to make decisions, etc. It's written in natural language so the testers can see what kind of information I'll be able to see, and to make it easier for me to read.

That sounds really amazing! :0 I'm not programming-minded so I'm not surprised I never thought to do something like that. Would you be willing to post your code/a tutorial to the cookbook? Or is it basic enough that someone with no programming background could learn (like me)? Thank you for posting your response. It was very helpful!

Sure. I actually got the idea from RenpyTom. Remind me next time you see me in #renpy. I know I'll forget otherwise.
I'll try to remember when I get home.

I am a professional Development Tester, so if you'd like I can send you some advice via PM about how to conduct successful test plans and what to ask your testers?

Vialixia wrote:I am a professional Development Tester, so if you'd like I can send you some advice via PM about how to conduct successful test plans and what to ask your testers?

Hi! Thank you for the offer! I think it would be more beneficial for others if you could post what you know here as well, but I understand if you need to keep details on the down-low. If it's better for you not to publicize the information I'm open to exchanging PMs with you. In the meantime, would you mind answering my questions for testers as if you were going to volunteer to test a game? (You don't have to answer everything since I know I asked quite a few questions >__<)

I have yet to have any experience with distributing my own games to testers, so I'll go ahead and answer your questions for testers.

What do you look for in an alpha/beta VN that makes you want to test it as a volunteer?
Basically the same things I look for in finished VN games I play; pretty or interesting art (which is not to say it has to be finished in a beta or alpha test version, rough sprites / BGs / placeholder CGs are all totally fine - the prettyness in this case has more to do with the quality level of the aesthetic vision of the game to me), interesting (or at least not completely overdone and cliché) story, and well developed characters. The most important of these, to me, is probably the characters. I can live with not very well-developed and/or cliché plots if the characters are sympathetic enough

Are you a completionist? If not, would you be willing to play until 100% completion for a testing period?
I'm not entirely sure what this question means (I'm not familiar with what a completionist is). Could you rephrase?

How do you feel about testing a certain route vs being free to test the whole thing?
It depends on how many routes there are. If there are a heap of different choices leading to different things, then it might be less confusing to focus on one main route. However, with a less complicated plot branching, then testing the entire game makes sense.

Would you prefer a feedback form to fill out or just to tell the developer your general feelings/critique of the game?
I think a combination of both could be helpful - a feedback form ensures that I don't overlook any aspects the developer really wants feedback on, but I would like to also give general feelings/critique on other things I might think of.

What mode of communication do you feel is best for you to give feedback to the dev? (e.g. PM, e-mail, form, live chat, etc)
I prefer to send an e-mail, and if the dev has additional questions or wants to follow up on something I mentioned then perhaps a live-chat as well.

How many builds would you be willing to test as a volunteer? (e.g. only alpha versions, only beta versions, both, just once, etc)
I'm combining the answer of this question with the next one, as they're closely connected for me.

How much time would you be willing to spend testing a game? (hours? days? until completion/release?)
This is very hard to answer - it depends on the size of the game, as well as how complete it is and how many routes I would test.
But basically, I would probably want to go over the story (the single route or all of them) at least once initially (as if I were playing the finished game), then once again after possible bug fixes etc.

Reikun wrote:Would you be willing to post your code/a tutorial to the cookbook? Or is it basic enough that someone with no programming background could learn (like me)?

Nothing too exciting, but hopefully it's readable enough to get the general idea: http://lemmasoft.renai.us/forums/viewto ... 51&t=21451

Vilhelmina wrote:Are you a completionist? If not, would you be willing to play until 100% completion for a testing period?
I'm not entirely sure what this question means (I'm not familiar with what a completionist is). Could you rephrase?

A "completionist" is someone who plays through VNs to get all the endings/unlock all bonuses (and maybe even see all dialogue variations). Basically someone who wants to fully complete the game vs someone who is satisfied getting only one or two endings.

Thank you for your responses! I'm actually not sure where my game is going to fall on the branching continuum yet so I'm glad to have your insight on testing routes vs. whole game and time investment on testing.

TrickWithAKnife wrote:

Reikun wrote:Would you be willing to post your code/a tutorial to the cookbook? Or is it basic enough that someone with no programming background could learn (like me)?

Nothing too exciting, but hopefully it's readable enough to get the general idea: http://lemmasoft.renai.us/forums/viewto ... 51&t=21451

Thank you so much! I will give it a shot as soon as I can! *U*