Closed Alpha/Beta testing

A place to discuss things that aren't specific to any one creator or game.
Forum rules
Ren'Py specific questions should be posted in the Ren'Py Questions and Annoucements forum, not here.
Post Reply
Message
Author
User avatar
Reikun
Miko-Class Veteran
Posts: 565
Joined: Tue Dec 20, 2011 9:57 pm
Completed: Mnemonic Devices, Ciikos Bridge, Helena's Flowers, The Madness
Projects: Fox in the Hollyhocks
Organization: skyharborr
itch: skyharborr
Contact:

Closed Alpha/Beta testing

#1 Post by Reikun »

I have a whole lot of questions about interacting with testers and running closed tests since I'm gunning for an alpha build for a personal project sometime in June. If anyone with experience running closed alpha/beta testing OR anyone who has done closed alpha/beta testing for VNs could answer any of them I'd be really grateful.

Distribution:
How do you distribute your game to your testers?
  • Do you send the game folders and they open and test from the Renpy launcher?
    Do you build distributions and make those available for DL?
If you distribute builds, do you archive any of the files or keep them open?

Feedback:
How do you gather feedback from your testers?
  • Do you ask them to answer specific questions or just ask for general impressions/feedback?
    Do you provide a form they can fill out or just send them a list of questions?
What methods of gathering feedback seems to work best? Worst?

For Devs and Testers:
What is a reasonable amount of time for an alpha/beta testing period to run if the testers are volunteers?
How many versions per testing period is reasonable?

If you have run an alpha/beta test:
What are some things to avoid when asking for people to test for you (esp. volunteers)?
How do you avoid "bad" testers? (e.g. People who say they want to test but don't deliver feedback)
Do you split up testing into storylines/paths so that Test Group A only plays Anna's route and Test Group B only plays Lily's route?
For longer games, how much do you expect testers to test? Only one route? The whole thing? As much as they can?

For people who test VNs/are interested in testing VNs:
What do you look for in an alpha/beta VN that makes you want to test it as a volunteer?
Are you a completionist? If not, would you be willing to play until 100% completion for a testing period?
How do you feel about testing a certain route vs being free to test the whole thing?
Would you prefer a feedback form to fill out or just to tell the developer your general feelings/critique of the game?
What mode of communication do you feel is best for you to give feedback to the dev? (e.g. PM, e-mail, form, live chat, etc)
How many builds would you be willing to test as a volunteer? (e.g. only alpha versions, only beta versions, both, just once, etc)
How much time would you be willing to spend testing a game? (hours? days? until completion/release?)

As you've probably noticed, I'm mostly wondering about dealing with volunteer testers. Would love to hear people's opinions/answers.
ImageImageImage

fastest way to contact me: DM @skyharborr on twitter

TrickWithAKnife
Eileen-Class Veteran
Posts: 1261
Joined: Fri Mar 16, 2012 11:38 am
Projects: Rika
Organization: Solo (for now)
IRC Nick: Trick
Location: Tokyo, Japan
Contact:

Re: Closed Alpha/Beta testing

#2 Post by TrickWithAKnife »

I'll touch on a few points from my limited experience.

Distribute however you like.
It's up to you if you want testers to be able to view files or not. If you are unsure, archive them first.

It's good to have different people as testers. People with a lot of experience with the genre, people who are kind of familiar, and some who generally have no idea what they are doing. You'll get different kinds of feedback from each group, all of it valuable.

It couldn't hurt to give them criteria for feedback, but also allow them to say what they like. Make sure to let them know how direct you want them to be. Be honest, so you don't end up getting your feeling hurt.

Personally I've added a feedback feature to my game (just for test versions), so the testers can click the Generate Feedback button, and it'll create a lot of useful information on how they played, and save it to a text file that they can email to me. That way I have a lot of detailed information that the testers don't need to write, leaving them free to write about other things.

The kind of information in the text file includes choices they've made, how long they took to read text at different stages of the game (to see where they got bored), how long they took to make decisions, etc. It's written in natural language so the testers can see what kind of information I'll be able to see, and to make it easier for me to read.
"We must teach them through the tools with which they are comfortable."
The #renpy IRC channel is a great place to chat with other devs. Due to the nature of IRC and timezone differences, people probably won't reply right away.

If you'd like to view or use any code from my VN PM me. All code is freely available without restriction, but also without warranty or (much) support.

User avatar
Reikun
Miko-Class Veteran
Posts: 565
Joined: Tue Dec 20, 2011 9:57 pm
Completed: Mnemonic Devices, Ciikos Bridge, Helena's Flowers, The Madness
Projects: Fox in the Hollyhocks
Organization: skyharborr
itch: skyharborr
Contact:

Re: Closed Alpha/Beta testing

#3 Post by Reikun »

TrickWithAKnife wrote:Personally I've added a feedback feature to my game (just for test versions), so the testers can click the Generate Feedback button, and it'll create a lot of useful information on how they played, and save it to a text file that they can email to me. That way I have a lot of detailed information that the testers don't need to write, leaving them free to write about other things.

The kind of information in the text file includes choices they've made, how long they took to read text at different stages of the game (to see where they got bored), how long they took to make decisions, etc. It's written in natural language so the testers can see what kind of information I'll be able to see, and to make it easier for me to read.
That sounds really amazing! :0 I'm not programming-minded so I'm not surprised I never thought to do something like that. Would you be willing to post your code/a tutorial to the cookbook? Or is it basic enough that someone with no programming background could learn (like me)? Thank you for posting your response. It was very helpful!
ImageImageImage

fastest way to contact me: DM @skyharborr on twitter

TrickWithAKnife
Eileen-Class Veteran
Posts: 1261
Joined: Fri Mar 16, 2012 11:38 am
Projects: Rika
Organization: Solo (for now)
IRC Nick: Trick
Location: Tokyo, Japan
Contact:

Re: Closed Alpha/Beta testing

#4 Post by TrickWithAKnife »

Sure. I actually got the idea from RenpyTom. Remind me next time you see me in #renpy. I know I'll forget otherwise.
I'll try to remember when I get home.
Last edited by TrickWithAKnife on Wed May 15, 2013 6:45 am, edited 1 time in total.
"We must teach them through the tools with which they are comfortable."
The #renpy IRC channel is a great place to chat with other devs. Due to the nature of IRC and timezone differences, people probably won't reply right away.

If you'd like to view or use any code from my VN PM me. All code is freely available without restriction, but also without warranty or (much) support.

User avatar
Vialixia
Regular
Posts: 111
Joined: Sat Feb 16, 2013 8:27 am
Location: UK
Contact:

Re: Closed Alpha/Beta testing

#5 Post by Vialixia »

I am a professional Development Tester, so if you'd like I can send you some advice via PM about how to conduct successful test plans and what to ask your testers?

User avatar
Reikun
Miko-Class Veteran
Posts: 565
Joined: Tue Dec 20, 2011 9:57 pm
Completed: Mnemonic Devices, Ciikos Bridge, Helena's Flowers, The Madness
Projects: Fox in the Hollyhocks
Organization: skyharborr
itch: skyharborr
Contact:

Re: Closed Alpha/Beta testing

#6 Post by Reikun »

Vialixia wrote:I am a professional Development Tester, so if you'd like I can send you some advice via PM about how to conduct successful test plans and what to ask your testers?
Hi! Thank you for the offer! I think it would be more beneficial for others if you could post what you know here as well, but I understand if you need to keep details on the down-low. If it's better for you not to publicize the information I'm open to exchanging PMs with you. In the meantime, would you mind answering my questions for testers as if you were going to volunteer to test a game? :) (You don't have to answer everything since I know I asked quite a few questions >__<)
ImageImageImage

fastest way to contact me: DM @skyharborr on twitter

User avatar
Vilhelmina
Regular
Posts: 37
Joined: Fri Apr 20, 2012 6:22 am
Organization: Cassis Connection Studios
Location: Sweden
Contact:

Re: Closed Alpha/Beta testing

#7 Post by Vilhelmina »

I have yet to have any experience with distributing my own games to testers, so I'll go ahead and answer your questions for testers. :)

What do you look for in an alpha/beta VN that makes you want to test it as a volunteer?
Basically the same things I look for in finished VN games I play; pretty or interesting art (which is not to say it has to be finished in a beta or alpha test version, rough sprites / BGs / placeholder CGs are all totally fine - the prettyness in this case has more to do with the quality level of the aesthetic vision of the game to me), interesting (or at least not completely overdone and cliché) story, and well developed characters. The most important of these, to me, is probably the characters. I can live with not very well-developed and/or cliché plots if the characters are sympathetic enough :D

Are you a completionist? If not, would you be willing to play until 100% completion for a testing period?
I'm not entirely sure what this question means (I'm not familiar with what a completionist is). Could you rephrase?

How do you feel about testing a certain route vs being free to test the whole thing?
It depends on how many routes there are. If there are a heap of different choices leading to different things, then it might be less confusing to focus on one main route. However, with a less complicated plot branching, then testing the entire game makes sense.

Would you prefer a feedback form to fill out or just to tell the developer your general feelings/critique of the game?
I think a combination of both could be helpful - a feedback form ensures that I don't overlook any aspects the developer really wants feedback on, but I would like to also give general feelings/critique on other things I might think of.

What mode of communication do you feel is best for you to give feedback to the dev? (e.g. PM, e-mail, form, live chat, etc)
I prefer to send an e-mail, and if the dev has additional questions or wants to follow up on something I mentioned then perhaps a live-chat as well.

How many builds would you be willing to test as a volunteer? (e.g. only alpha versions, only beta versions, both, just once, etc)
I'm combining the answer of this question with the next one, as they're closely connected for me.

How much time would you be willing to spend testing a game? (hours? days? until completion/release?)
This is very hard to answer - it depends on the size of the game, as well as how complete it is and how many routes I would test.
But basically, I would probably want to go over the story (the single route or all of them) at least once initially (as if I were playing the finished game), then once again after possible bug fixes etc.
Artist/writer by night, MD/PhD student by day
Looking for medical fact checking / advice for your game? PM me!

art gallery: Vilhelmina @ deviantART | Founder of Cassis Connection Studios

TrickWithAKnife
Eileen-Class Veteran
Posts: 1261
Joined: Fri Mar 16, 2012 11:38 am
Projects: Rika
Organization: Solo (for now)
IRC Nick: Trick
Location: Tokyo, Japan
Contact:

Re: Closed Alpha/Beta testing

#8 Post by TrickWithAKnife »

Reikun wrote:Would you be willing to post your code/a tutorial to the cookbook? Or is it basic enough that someone with no programming background could learn (like me)?
Nothing too exciting, but hopefully it's readable enough to get the general idea: http://lemmasoft.renai.us/forums/viewto ... 51&t=21451
"We must teach them through the tools with which they are comfortable."
The #renpy IRC channel is a great place to chat with other devs. Due to the nature of IRC and timezone differences, people probably won't reply right away.

If you'd like to view or use any code from my VN PM me. All code is freely available without restriction, but also without warranty or (much) support.

User avatar
Reikun
Miko-Class Veteran
Posts: 565
Joined: Tue Dec 20, 2011 9:57 pm
Completed: Mnemonic Devices, Ciikos Bridge, Helena's Flowers, The Madness
Projects: Fox in the Hollyhocks
Organization: skyharborr
itch: skyharborr
Contact:

Re: Closed Alpha/Beta testing

#9 Post by Reikun »

Vilhelmina wrote:Are you a completionist? If not, would you be willing to play until 100% completion for a testing period?
I'm not entirely sure what this question means (I'm not familiar with what a completionist is). Could you rephrase?
A "completionist" is someone who plays through VNs to get all the endings/unlock all bonuses (and maybe even see all dialogue variations). Basically someone who wants to fully complete the game vs someone who is satisfied getting only one or two endings. :)

Thank you for your responses! I'm actually not sure where my game is going to fall on the branching continuum yet so I'm glad to have your insight on testing routes vs. whole game and time investment on testing.
TrickWithAKnife wrote:
Reikun wrote:Would you be willing to post your code/a tutorial to the cookbook? Or is it basic enough that someone with no programming background could learn (like me)?
Nothing too exciting, but hopefully it's readable enough to get the general idea: http://lemmasoft.renai.us/forums/viewto ... 51&t=21451
Thank you so much! I will give it a shot as soon as I can! *U*
ImageImageImage

fastest way to contact me: DM @skyharborr on twitter

Post Reply

Who is online

Users browsing this forum: Semrush [Bot]