[solved] Persistent dictionary disappearing upon restart :(

[goldo]

Hi everyone,

I have been struggling for the past few days with a very strange issue with Ren'py not properly saving/restoring a persistent dictionary.

I was trying to implement a dynamic CG gallery to display images that are not declared as ren'py images (I won't get into the various reasons for this here, but the idea is that players can use their own image packs).

Those images are shown within specific screens, and in turn those screens track the file path for the images that have been seen as 'True' in a persistent dictionary. This more or less emulates the way renpy.seen_image() works for regular renpy images.

Everything works fine and dandy as long as Ren'py is running. Images unlock and display all right in the CG Gallery, and the dictionary works fine. A 'renpy.full_restart()' works just fine too.


However, when closing and restarting the game or doing a 'renpy.utter_restart()', all hell breaks loose.


The persistent dictionary disappears, all stored values are removed and the CG gallery is empty. I narrowed the problem down to the persistent dictionary itself disappearing: the code for the gallery seems not to matter. It's not just the individual values that are reset, the dictionary itself is gone.

On a few occasions, when only a handful of pictures were unlocked, the dictionary saved properly. But that was a very rare case. In most cases, the dict simply resets. I strongly suspect that something is messing with Ren'py's system for saving and/or restoring persistent values, because on occasion other persistent values from the game (such as the 'persistent.seen_intro' value I use to track a new game) have gone missing as well, even though they share zero code with the offending script. Those errors, if they happen, go completely silent.

It does not seem to be related to the method I use to declare the dict. I have tried using :

Code: Select all

if persistent.seen_dict is None:
	persistent.seen_dict = defaultdict(bool)
	

With exactly the same (lack of) result.

I also don't think it's because I am using a DefaultDict: I am using other DefaultDicts as persistent variables in the game (for some menu options) without any trouble. I have also tried variants with a regular dictionary, or even just a plain list: the same bug happens either way.

I should also mention that I have used the 'Delete persistent' option in the Ren'py launcher to make sure previous attempts didn't mess with the persistent data. I have also played around with the init order, but didn't notice any difference.

So, I'm at a complete loss to explain what is going on here. Things that could maybe play a role (but I am not sure how to work around them right now):
- The image paths are long: The image file paths used as keys for the dictionary are strings that can be quite long and run over, say, 60 or 70 characters. I'm not sure why that would be a problem for a modern system, but, there.
- The number of images can be large: several hundreds. Does that somehow 'overload' the dictionary with too many keys? I'm not sure, but the fact that it doesn't reset when only a few pictures are unlocked could suggest that.
- The image paths contain '/' characters: Again, I'm not sure why it would mess with Ren'py saving persistent variables, but that's the only specificity I could think of compared to the other dictionaries I use.

The code for the game itself is very complex and largely unrelated, but I have included the relevant bits below. I have double-checked the whole code, and nothing else messes up with or even accesses the dictionary in question.

If anyone can help, you are my only Hope.

Code: Select all

init -999 python:
    from collections import defaultdict
    
init -3:
    define persistent.seen_dict = defaultdict(bool)

init -2 python:

    IMGFORMATS = (".jpg", ".jpeg", ".png",".bmp", ".gif")
    
    def is_imgfile(file):
        if (file[-4:] in IMGFORMATS or file[-5:] in IMGFORMATS):
            return True
        return False
    
    ## This method checks if an image has been seen using the persistent dictionary or the regular renpy.seen_image. 
    ## It works fine as long as the game is running.
    def was_seen(pic): # Where pic is a String, either a file path or a renpy image name
        
        if is_imgfile(pic): # 
            if persistent.seen_dict[pic]:
                return True
        
        elif renpy.seen_image(pic) or persistent.seen_dict[pic]:
            return True
        
        return False

    def list_girl_packs(): # Returns a list of girl pack folders. Used for CG gallery
        
        pack_list = []
        
        for file in [f for f in renpy.list_files() if f.startswith("girls/")]:
                
        file_parts = file.split("/")
            
         if file_parts[1] not in pack_list:
                pack_list.append(file_parts[1])
                
        return pack_list


    def init_galleries(): # This is run on start-up in an init block. The bug doesn't seem to originate from the gallery code itself, even though it's unwieldy.

        global gp_gallery
        
        gp_gallery = defaultdict(bool)
        
        for pack in list_girl_packs():
            gp_gallery[pack] = Gallery()
            gp_gallery[pack].pics = []  # Not a native parameter for the Gallery object, used here for convenience
            
            gp_gallery[pack].slideshow_delay = 1.0
            gp_gallery[pack].navigation=True
            gp_gallery[pack].span_buttons=True
            gp_gallery[pack].locked_button=lock
            
            for file in [f for f in renpy.list_files() if f.startswith("girls/" + pack) and is_imgfile(f)]:
                gp_gallery[pack].button(file)
                gp_gallery[pack].condition("was_seen('" + file + "')")
                gp_gallery[pack].image(file)
                gp_gallery[pack].pics.append(file)
                
            gp_gallery[pack].renpy_images = False


#### SCREENS ####

## This screen is used to display non-renpy pictures
screen show_event(event_pic, x = None, y = None, proportional = True, bg = "#000"):
    
    tag show_event
    
    zorder 0
    
    frame:
        background bg

        if event_pic:
            $ persistent.seen_dict[event_pic.path] = True # This is how the game tracks that this particular picture has been seen.
            add event_pic.get(x, y, proportional) xalign 0.5 yalign 0.0


## The gallery screen displays the gallery attached to a particular girl pack. It is called from the main menu.
screen gallery(name="", gal=None, bg=None): # The Gallery object must have a pics variable (a list of renpy displayables or image files)
    
    tag menu
    
    default page=0
    
    key "mouseup_3" action Return()
    key "K_ESCAPE" action Return()
    
    add "#000"
    add bg xalign 0.5 yalign 0.5
    
    hbox:

        vbox spacing 10:

            text name

            frame background None:

                id "gallery"

                xsize 1.0
                ysize 0.9

                has hbox box_wrap True

                $ index = page*12

                for i in xrange(12):

                    if index+i < len(gal.pics):
                        $ pic = gal.pics[index+i]

                        frame background None xsize 250 ysize 190:
                            if gal.renpy_images:
                                add gal.make_button(pic, ProportionalScale(ImageReference(pic), 240, 180), xalign=0.5) alpha 0.8
                            else:
                                add gal.make_button(pic, ProportionalScale(pic, 240, 180), xalign=0.5, background=c_cream) alpha 0.8


            $ max_page = (len(gal.pics)-1) // 12

            text "Page " + str(page+1) + "/" + str(max_page+1) size 14

            hbox:
    #            textbutton "Start Slideshow" action gal.ToggleSlideshow()

                if page > 0:
                    key "K_LEFT" action SetScreenVariable("page", page-1)
                    key "repeat_K_LEFT" action SetScreenVariable("page", page-1)
                    textbutton "Previous" action SetScreenVariable("page", page-1)
                if page < max_page:
                    key "K_RIGHT" action SetScreenVariable("page", page+1)
                    key "repeat_K_RIGHT" action SetScreenVariable("page", page+1)
                    textbutton "Next" action SetScreenVariable("page", page+1)
                    
                key "K_HOME" action SetScreenVariable("page", 0)
                key "K_END" action SetScreenVariable("page", max_page)
                
                textbutton "Return" action Return()


## These two screens override the native ren'py screens
screen _gallery:

    if locked:
        add "#000"
        text _("Image [index] of [count] locked.") align (0.5, 0.5)
    else:
        add "#000"
        
        for d in displayables:
            add d xalign 0.5 yalign 0.0

    if gallery.slideshow:
        timer gallery.slideshow_delay action Return("next") repeat True

    key "game_menu" action gallery.Return()

    if gallery.navigation:
        use gallery_navigation
        
        
screen gallery_navigation:
    hbox:
        spacing 20

        style_group "gallery"
        align (.98, .98)
        
        key "mouseup_3" action gallery.Return()
        key "K_ESCAPE" action gallery.Return()
        key "K_LEFT" action gallery.Previous(unlocked=True)
        key "repeat_K_LEFT" action gallery.Previous(unlocked=True)
        key 'K_BACKSPACE' action gallery.Previous(unlocked=True)
        key "K_RIGHT" action gallery.Next(unlocked=True)
        key "repeat_K_RIGHT" action gallery.Next(unlocked=True)
        key 'K_SPACE' action gallery.Next(unlocked=True)
        key "K_RETURN" action gallery.ToggleSlideshow()
        key "K_SCROLLOCK" action gallery.ToggleSlideshow()
        
        textbutton _("prev") action gallery.Previous(unlocked=True)
        textbutton _("next") action gallery.Next(unlocked=True)
        textbutton _("slideshow") action gallery.ToggleSlideshow()
        textbutton _("return") action gallery.Return()

    python:
        style.gallery = Style(style.default)
        style.gallery_button.background = None
        style.gallery_button_text.color = "#666"
        style.gallery_button_text.hover_color = "#fff"
        style.gallery_button_text.selected_color = "#fff"
        style.gallery_button_text.size = 16

[kivik]

Try

Code: Select all

    if hasattr(persistent, "seen_dict"):
	persistent.seen_dict = defaultdict(bool)

As you said, it may not be the problem, but my understanding is that you check if an attribute exists in an object by using hasattr().

I can't run your code as it without the rest of the scripts so I can't test what's going on myself unfortunately

[xavimat]

I don't know if this will work with defaultdict, but, try using "define" for persistent variables and see if this works. Use it outside any "init" block.

Code: Select all

define persistent.seen_dict = {}

[kivik]

I don't know if this will work with defaultdict, but, try using "define" for persistent variables and see if this works. Use it outside any "init" block.

Code: Select all

define persistent.seen_dict = {}

Would that not reset the dict each time on load? I thought we wouldn't want to do that to persistent variables?

[Ocelot]

No, that would not reset it. define has a special handling for persistent objects (also persistent object has every attribute possible with value "None" by defining __getattr__ magic method)

[goldo]

Hi guys! Thanks for your replies.

I am using define since it is the recommended method (see the code part in the OP):

Code: Select all

init -3:
    define persistent.seen_dict = defaultdict(bool)

Before calling for help, I did try other methods to see if the defaultdict was causing problems.

I have tried both using a regular dict like this:

Code: Select all

init -3:
    define persistent.seen_dict = {}

Or even a plain list:

Code: Select all

init -3:
    define persistent.seen_dict = []

Of course adjusting the other coding so that it would work as intended.

In both cases, the same bug happened. So it's not only the defaultdict that causes this: Any persistent variable I use for this particular task disappears in the same fashion. It's very frustrating.


Edit: Reading your post carefully, I haven't tried using define outside of an init block. I will and report here.
Edit2: Nope, same old same old. Even when defined outside of an init block, the variable still resets

[philat]

At this point, probably better to share your project (or recreate the bug with placeholder assets).

[goldo]

Ok, so I have made some progress diagnosing the issue (but still couldn't solve it).

The problem lies with the screen:

Code: Select all

## This screen is used to display non-renpy pictures
screen show_event(event_pic, x = None, y = None, proportional = True, bg = "#000"):
    
    tag show_event
    
    zorder 0
    
    frame:
        background bg

        if event_pic:
            $ persistent.seen_dict[event_pic.path] = True # This is how the game tracks that this particular picture has been seen.
            add event_pic.get(x, y, proportional) xalign 0.5 yalign 0.0

Specifically this line:

Code: Select all

$ persistent.seen_dict[event_pic.path] = True # This is how the game tracks that this particular picture has been seen.

It seems that setting the value within the screen like this is what kills the dictionary or list. I get the same error if I use a custom function like '$ unlock_pic()' within the screen instead of setting the value directly.

I have tried using other methods to unlock pictures and they all work fine, even for a large number of pictures. So definitely the problem is with this screen.

My problem is that the 'show_event' screen is called from a very large number of lines in the code, so if I have to add more code to unlock pictures whenever it's called it's going to be a lot of clutter. Is there a safe way to change a variable upon calling a screen?

[Ocelot]

Can you share code for unlock_pic function you used? I have a suspiction that it has something to do with variable scope.

[goldo]

Sure, actually I was using that function with a different version of the script (the one that uses a plain list), but the principle is the same. It looked like this:

Code: Select all

def unlock_pic(pic):
	if pic not in persistent.seen_list:
		persistent.seen_list.append(pic)

And of course, the screen code was then:

Code: Select all

## This screen is used to display non-renpy pictures
screen show_event(event_pic, x = None, y = None, proportional = True, bg = "#000"):
    
    tag show_event
    
    zorder 0
    
    frame:
        background bg

        if event_pic:
            $ unlock_pic(event_pic) # This is how the game tracks that this particular picture has been seen.
            add event_pic.get(x, y, proportional) xalign 0.5 yalign 0.0

[kivik]

Just to check - you've got persistent.seen_list here, but your gallery uses persisent.seen_dict - are you using the same in both codes?

[goldo]

Yes, that's why I mentioned this code is from the list variant (I tried using a defaultdict, a normal dict or a list). I wanted to make sure the problem wasn't caused by the defaultdict.

I haven't tried using a function with the defaultdict variant, but I don't believe it would make a difference. I'll see if I can reproduce the bug in a test game, that would be easier to share.

[kivik]

Ok, I meant the attribute / variable name not the data type though, but I assume that's been factored in as well from your last reply. Sorry for not being able to help on this one!

[philat]

Besides the fact that screens aren't supposed to run code that has effects outside the screen -- Seriously. Share something that can be recreated.

[goldo]

So, there's a simple game that goes some way into demonstrating the problem I'm having:
http://www.mediafire.com/file/ybw8pea51 ... 1.0-pc.zip

You can start it, click on the menu to display a picture, and watch the persistent variable evolve (in this case, a list).

It's pretty clear multiple items get added to the list every time, not just the picture that was clicked on. The list gets cluttered very quickly, and there are only 3 pictures. With hundreds of pictures in the actual game, I'm guessing the combinations get out of hand quickly, and maybe the list exceeds the available memory for storing persistent variables, or something.

The problem comes from this part:

Code: Select all

## This screen is used to display non-renpy pictures
screen show_pic(pic, x = None, y = None, proportional = True, bg = "#000"):
    
    tag menu
    
    modal True
    
    zorder 10
    
    add bg

    if pic:
        $ persistent.seen_list.append(pic) # This is how the game tracks that this particular picture has been seen.
        add Picture(pic).get(x, y, proportional) xalign 0.5 yalign 0.5
            
    key "mouseup_1" action Return()

If I had to guess, I'd say Ren'py predicting screens is what's causing this. I'm not 100% sure this is the whole problem, but at least it shows that my idea of handling the unlocking through the screen itself was bad.

Unfortunately, short of adding additional code every time an instance of 'show screen' is called (and that's over a hundred and counting), I cannot easily go around this (not to mention it would be easy to forget and make a mistake when writing new events).

Is there a way to safely influence a persistent variable when calling a screen?

Edit: My suspicions are confirmed: it turns out that when calling the screen with 'nopredict', the screen behaves as intended. I didn't see a noticeable performance difference but I'm sure it's there. If anyone knows a better way, I'd love to learn...