Android: Incremental builds

In this forum we discuss the future of Ren'Py, both bug fixes and longer-term development. Pre-releases are announced and discussed here.
Post Reply
Message
Author
Apa
Regular
Posts: 103
Joined: Fri Dec 23, 2016 2:26 am
Location: NYC
Contact:

Android: Incremental builds

#1 Post by Apa »

Guys,
I think, I have an idea how do incremental builds for Android. :idea:
It can be "Update package" item in Android menu.

Steps:
Checks if apk (and obb, if expansion is selected) exists in rapt\bin folder.
I’ll refer to apk or obb as zip [file].

For update we need to build a list of updated files.
Walk zip file tree and compare files timestamps to corresponding source game folder files.
If the latter is newer - add it to the list.
Use the list to delete files from archive in one pass.
If zip is APK: delete signatures in same pass.
Then use the list to update archive (without intermediate copy as 'x-*' files):

Code: Select all

zipfile.write('C:\renpy-6.99.12.2-sdk\the_question\game\script.rpyc', arcname='assets\x-game\x-script.rpyc')
* The process may vary of course, but the main idea is NOT to rebuild everything from scratch.
If zip is APK: resign + zipalign.

This should make Android builds a lot faster.
AFAIK, very few people do Android builds, but those who do know it’s quite slow process.
Again, it’s needed because RenPy game behavior in emulator and real device is not the same and having fast builds should make Android development more pleasant.

I guess, very similar process can be use for initial APK build (without intermediate copying as well), but it’s not really needed.
Once we have incremental build - we can do initial build once only.

I do know, v7 is underway, so it’s not a request of any sort.
Just want to throw an idea for discussion.

User avatar
PyTom
Ren'Py Creator
Posts: 16088
Joined: Mon Feb 02, 2004 10:58 am
Completed: Moonlight Walks
Projects: Ren'Py
IRC Nick: renpytom
Github: renpytom
itch: renpytom
Location: Kings Park, NY
Contact:

Re: Android: Incremental builds

#2 Post by PyTom »

Hm... would it make sense to just do this with the obb? Or simply rebuild the obb and not the apk if nothing has changed? I really really don't want to implement an unofficial android build process.
Supporting creators since 2004
(When was the last time you backed up your game?)
"Do good work." - Virgil Ivan "Gus" Grissom
Software > Drama • https://www.patreon.com/renpytom

Apa
Regular
Posts: 103
Joined: Fri Dec 23, 2016 2:26 am
Location: NYC
Contact:

Re: Android: Incremental builds

#3 Post by Apa »

PyTom wrote:Hm... would it make sense to just do this with the obb?
Yep, that’ll do just fine!
PyTom wrote:Or simply rebuild the obb and not the apk if nothing has changed?
If we choose apk with expansion file (obb), updating obb will do just fine!
However, rebuilding obb "from the scratch" makes less sense, as we need essentially recompress all files.
And since most of them didn’t change - why waste CPU / energy / our time to do redundant job?
Deleting files from zip is really easy.
PyTom wrote:I really really don't want to implement an unofficial android build process.
You don’t have to (and there is nothing special in it either)
I was talking only about updating files related to game itself, specifically in assets/x-game and assets/x-renpy folders.
They can be updated same way obb is.
The only extra step and to sign & zip-align APK again.
But all that "fast APK update" not really needed, rebuilding obb alone gives us incremental builds already!


There is one thing I’m not sure how to handle properly.
I don’t even know how to describe it...
Basically, when obb is updated - app needs to be terminated, otherwise it’ll try to use old references into new obb. I'm not even sure, this is what causing problem in first place.
May be if app can detect, obb timestamp has changed - app needs to close and open obb again (and flush cache if there is any?)


PyTom,
Unrelated question:
What is the purpose of \assets\private.mp3 file? :?:
It seems changed on every APK built?

Apa
Regular
Posts: 103
Joined: Fri Dec 23, 2016 2:26 am
Location: NYC
Contact:

Re: Android: Incremental builds

#4 Post by Apa »

I was working on "update obb prototype" and found that after python 2.4 there is NO way to delete a file in zip without decompress/recompressing for all remaining files...

This is very disappointing, but not a show stopper at any rate.
I can see 2 workarounds as of now:
1. Use external zip program to delete old files and python to add newer ones with x- names. The ovious drawback is we need 2 passes (1 to delete files and another for add them)

2. Tweak existing "odd build" function:
To make obb/zip current approach is to copy all required files with x- names into directory tree (for zipping).
The idea is NOT to copy files, but create hardlinks (with x- name) instead.
Creating links costs nothing (comparing to copy same files)
There resulting directory tree can be used either to create or update OBB.

I can certainly do prototype myself, just want check with you guys first.
Does anybody see any flaws with #1 or #2 or have better idea how to tackle it?

User avatar
PyTom
Ren'Py Creator
Posts: 16088
Joined: Mon Feb 02, 2004 10:58 am
Completed: Moonlight Walks
Projects: Ren'Py
IRC Nick: renpytom
Github: renpytom
itch: renpytom
Location: Kings Park, NY
Contact:

Re: Android: Incremental builds

#5 Post by PyTom »

Hardlinks don't work on all platforms, do they?
Supporting creators since 2004
(When was the last time you backed up your game?)
"Do good work." - Virgil Ivan "Gus" Grissom
Software > Drama • https://www.patreon.com/renpytom

Apa
Regular
Posts: 103
Joined: Fri Dec 23, 2016 2:26 am
Location: NYC
Contact:

Re: Android: Incremental builds

#6 Post by Apa »

PyTom wrote:Hardlinks don't work on all platforms, do they?
Yep!
And not all FS supports them either. For windows need a "Windows XP" at least (and NTFS, obviously)...
Let me write a proof of concept first - seems like I'm the only one who need it so far. :oops:
No need to waste your time!

Apa
Regular
Posts: 103
Joined: Fri Dec 23, 2016 2:26 am
Location: NYC
Contact:

Re: Android: Incremental builds

#7 Post by Apa »

Progress so far:
I used

Code: Select all

def make_tree(src, dest):
from rapt\build.py as starting point for stand-alone prototype. Replaced line

Code: Select all

    shutil.copytree(src, dest, ignore=ignore)
with code copytree example with following replacements

Code: Select all

        srcname = os.path.join(src, name)
        dstname = os.path.join(dst, 'x-' + name)
            if os.path.isdir(srcname):
                copytree(srcname, dstname, symlinks, ignore)
            else:
                ntfsutils.hardlink.create(srcname, dstname)
It builds hard linked copy \assets\x-game tree (half of OBB)
Don’t know yet, how to build another half: \assets\x-renpy\x-common tree.
It has fewer files than corresponding C:\renpy-6.99.12.2-sdk\renpy\common folder.
Could you point me to exact procedure to complete prototype, please?

From another hand, I’ve noticed (finally!), by trying to avoid OBB re-compression I fight windmills:
There is no OBB compression (for performance reason, I guess)

Code: Select all

        def make_expansion():
            zf = zipfile.ZipFile(plat.path(expansion_file), "w", zipfile.ZIP_STORED)
I can see 3 approaches so far, pretty much no brainier which one to choose:
  1. Rebuild obb from scratch writing x- files without intermediate copy (as PyTom suggested)
  2. Build changed files list and create a obb copy via copying non-changed files from old obb and new from OS folders (No much faster than #1, probably, but does need some coding)
  3. Build hard-linked directory tree (with x- files). Use external zip program to refresh OBB.
    Would make sense, it OBB is created with compression, not much otherwise.

Post Reply

Who is online

Users browsing this forum: No registered users