How does the testcase system work?

[Phanes]

Ren'Py seems to have an internal testing system that allows for automated testing of a game, making choices as it goes: https://github.com/renpy/renpy/blob/mas ... tcases.rpy

Recognizing that this presumably isn't robust enough to show up in the documentation: How is this meant to be run?

I'd love to be able to run various scenarios against new versions of my game, maybe comparing the game history to previous "blessed" versions. My dream is to be able to create an automated test suite for every choice in my game that exports a series of transcripts that I can diff against previous results to determine if there are unexpected changes, recognizing that this might not surface graphical issues.

[zmook]

Its official support status seems to be somewhere around "PyTom admits it exists, but doesn't answer questions about it."

viewtopic.php?t=42208

[Phanes]

My experimentation so far suggests one can run them like renpy.py "<project directory>" test <testcase name>, but if I do so it just yields some OpenGL errors, presumably because I'm just running it straight with a local Python 2 install without any packages set up. I'm not sure if there's a better way to run that -- I don't see a venv or anything in the renpy directory, just some bare lib folders -- but I'm guessing that it is supposed to start up a visible instance of the project to run the testcase. It seems to run the "default" testcase if you don't specify one, and I don't see any support for test suites or running all tests (but I might be missing something).

If anyone who comes across this thread has had success with running tests on their own project, I'd love some insight.

[PyTom]

You should be able to run it with renpy.sh or renpy.exe and the rest of the command the same.

[Phanes]

You should be able to run it with renpy.sh or renpy.exe and the rest of the command the same.

Thank you so much! This let me figure out the rest. From what I can tell, the following is how to get testing working. It'll require some comfort with command lines and file systems, and experience with a test suite like Selenium will be handy for writing tests.

Create testcases anywhere in your project that represent a number of UX actions to take in your game. Follow https://github.com/renpy/renpy/blob/mas ... tcases.rpy as an example. One especially useful construction is click until "Certain Choice", which will click through dialogue until a choice appears with the given text, then click on that text. Also note how they use the test settings to cut transitions short.

You can have any number of testcases, and they can call each other. The testcase named default will be run if no name is specified on the command line.

To run the tests, you'll need a command line environment with the SDL_VIDEODRIVER variable set to something besides the default ("dummy") or it'll try to run the tests without opening a window and error out. For me, using Git Bash on Windows, this means I need to run export SDL_VIDEODRIVER=windows if I don't want to mess around with configuring variables elsewhere.

Finally, to run the tests, the command I've settled on is:

Code: Select all

renpy.exe --errors-in-editor <PROJECT DIR> test <TESTCASE NAME>

The --errors-in-editor isn't necessary, but will open a text file at the end if any errors occur.

When you finish your testcase, it will not automatically exit. The magic sequence that seems to reliably exit my game at the end, starting from an in-game dialogue situation, is:

Code: Select all

"History"
pause .5

"Main Menu"

"Yes"

"Quit"

Finally, something that I couldn't find in the existing testcases in the source is that you can include a line in your testcase like assert gender == GENDER_FEMALE and you'll get an error if the assertion isn't truthy.

If your test is successful, you'll see the window appear, run through the actions you've specified, and then close (if you manually exited as part of the test). If a text file with errors doesn't appear, you're golden!