APK full rebuild from source for F-Droid

Discuss how to use the Ren'Py engine to create visual novels and story-based games. New releases are announced in this section.
Forum rules
This is the right place for Ren'Py help. Please ask one question per thread, use a descriptive subject like 'NotFound error in option.rpy' , and include all the relevant information - especially any relevant code and traceback messages. Use the code tag to format scripts.
Post Reply
Message
Author
User avatar
bosinpai
Regular
Posts: 100
Joined: Sat Nov 22, 2014 9:11 am
Contact:

APK full rebuild from source for F-Droid

#1 Post by bosinpai » Thu Mar 26, 2015 11:17 am

Hi,

I'd like to add our Little Mauve Riding Hood game to F-Droid, which is an alternate store for Android dedicated to free & open source software - including games :)

To ensure the game really is free software, F-Droid rebuilds all the submitted apps: you don't send an .apk - instead you submit a recipe which may point to a git repository, and the final .apk is built on their server.

The following documentation tracks my progress towards this goal.
Currently there are issues with:
  • compilation in the limited fdroid buildserver environment, in particular rebuilding a full native renpy to start the APK builder
  • stubbing several renpy optional utilities interfacing with pre-built or proprietary libraries
  • missing Android SDK extras from Google

Code: Select all

# Clone the projects using this layout:
WD=/mnt/workdir/renpy
cd $WD
git clone https://github.com/renpy/renpy -b 6.99.0.303 renpy-git/
git clone https://github.com/renpy/rapt -b renpy-6.99.0.303 rapt-git/
git clone https://github.com/renpy/pygame_sdl2 -b renpy-6.99.0.303 renpy-git/pygame_sdl2/
git clone https://github.com/renpy/python-for-android -b renpy-6.99.0.303 rapt-git/python-for-android/

cd rapt-git/

ln -nfs $WD/android-ndk-r10c  # http://dl.google.com/android/ndk/android-ndk-r10c-linux-x86_64.bin
ln -nfs $WD/android-sdk-linux android-sdk-r23

# Build RAPT
# Work-arounds
ln -nfs android-sdk-r23 android-sdk
# compile all .pyo, 'build_renpy.sh' forgets e.g. glblacklist.pyo and swdraw.pyo
(cd $WD/renpy-git && python -O -m compileall .)
# Make it possible to compile without rebuilding a full native Ren'Py,
# which requires installing lots of -dev .deb (as root...):
# - sed -i -e 's|try ./run.sh the_question compile|#&|' build_renpy.sh
# - comment out references to angle in ../renpy-git/module/setup.py
# Optional: disable pre-built or proprietary APIs
# - sed -i -e '/ouya-sdk.*.jar/s/^/#/' build_renpy.sh
# - sed -i -e '/amazon-iap.*.jar/s/^/#/' build_renpy.sh
# - stub python-for-android/src/src/org/renpy/iap/*Store.java
#pip install -U cython==0.21.1
#sudo apt-get install cython  # 0.21.1 already pre-installed in the fdroid buildserver
LANG=C ./build_renpy.sh renpy $WD/renpy-git/ $WD/renpy-git/pygame_sdl2/

# Prepare RAPT distribution in dist/renpy/
cat <<EOF > local.properties
key.alias=android
key.store.password=android
key.alias.password=android
key.store=android.keystore
EOF
sed -i -e "s|^try ./build_renpy.sh|#&|" test_renpy.sh  # already done with non-'pytom' paths ;)
ln -nfs $WD/apache-ant-1.9.3
./test_renpy.sh
# ^^^ FYI installsdk installs APL-licensed extensions not present in the fdroid buildserver:
# (cd android-sdk-r23 && ./tools/android update sdk -u -f -a -t build-tools-20.0.0,android-19,platform-tools,extra-google-play_apk_expansion,extra-google-play_licensing)
rm -rf $WD/renpy-git/rapt/
cp -a dist/renpy $WD/renpy-git/rapt/


# Build the full Ren'Py to be able to run the APK builder..
# Cf. https://github.com/renpy/renpy/blob/master/README.rst:
sudo apt-get install python-dev libavcodec-dev libavformat-dev \
    libavresample-dev libswscale-dev libfreetype6-dev libglew1.6-dev \
    libfribidi-dev libsdl2-dev libsdl2-image-dev libsdl2-gfx-dev \
    libsdl2-mixer-dev libsdl2-ttf-dev libjpeg-turbo8-dev
export RENPY_DEPS_INSTALL="/usr::/usr/lib/x86_64-linux-gnu/"
sudo apt-get install virtualenvwrapper
mkvirtualenv renpy_android
(
  cd $WD/renpy-git/pygame_sdl2/
  # Fixes ~/.virtualenvs/renpy_android/include/python2.7 being a symlink to (non-writable) /usr/include/python2.7
  python fix_virtualenv.py $VIRTUAL_ENV
)
pip install -U cython==0.21.1
(cd ../renpy-git && ./run.sh the_question compile)

# Build the game APK
cd $WD/renpy-git/rapt/
# - edit ../launcher/game/distribute.rpy, remove calls to self.add_python(), self.add_mac_files(), self.add_windows_files()
# - cp your 'android.keystore' here
echo '{"layout": null, "name": "The Question", "package": "org.renpy.thequestion", "orientation": "sensor", "include_pil": false, "expansion": false, "source": false, "icon_name": "The Question", "version": "1.0", "target_version": 14, "permissions": ["VIBRATE"], "include_sqlite": false, "store": "none", "numeric_version": "100"}' > ../the_question/.android.json
python ../renpy.py ../launcher android_build ../the_question release
#python ../renpy.py ../launcher android_build --launch ../the_question release install


# Debug
./android-sdk-r23/platform-tools/adb pull /storage/sdcard0/Android/data/org.renpy.thequestion/files/log.txt
./android-sdk-r23/platform-tools/adb logcat
---

EDIT: here was my (obsolete) original post, the remaining issues are different:

Hi,

I'd like to add our Little Mauve Riding Hood game to F-Droid, which is an alternate store for Android dedicated to free & open source software - including games :)

To ensure the game really is free software, F-Droid rebuilds all the submitted apps: you don't send an .apk - instead you submit a recipe which may point to a git repository, and the final .apk is built on their server.

I'm trying to use ./build_renpy.sh from rapt-git, but there seem to be several implicit requirements.
Also there's no release tag for rapt-git and python_sdl2, so I don't know which revision I should checkout exactly.

Running I'm trying:

Code: Select all

$ apt-get install libsdl2-dev
$ PYTHONPATH=$(python -c 'import sys; print ":".join(sys.path)') bash -x ./build_renpy.sh android /mnt/workdir/renpy/renpy-6.99.0-git/ /mnt/workdir/renpy/pygame_sdl2/
...
'Could not find required library SDL2.'
I'll also need to run the APK build through the command line (rather than the renpy launcher).

Is there documentation on how to do a full APK rebuild from sources?
I would be happy to contribute.
Last edited by bosinpai on Mon Mar 30, 2015 5:29 pm, edited 10 times in total.

User avatar
PyTom
Ren'Py Creator
Posts: 15893
Joined: Mon Feb 02, 2004 10:58 am
Completed: Moonlight Walks
Projects: Ren'Py
IRC Nick: renpytom
Github: renpytom
itch: renpytom
Location: Kings Park, NY
Contact:

Re: APK full rebuild from source for F-Droid

#2 Post by PyTom » Thu Mar 26, 2015 12:07 pm

To some extent, if you want to do this, you're going to have to figure it out yourself. It's a somewhat silly requirement - just because something comes out of a git archive, doesn't mean it's open source. And you'd have to hack things a little to make a RAPT build fully open source, as by default we include at least some support for play, amazon, and ouya, all of which depends on closed-source software.

That being said, to build rapt, you need to have the android-ndk-r10c, android-sdk-r23, apache-ant-1.9.3, in the appropriate places underneath the rapt clone. You'll also want to set PYGAME_SDL2_ROOT and RENPYROOT To the paths to pygame_sdl2 and Ren'Py, respectively.

If you take a look at test_renpy.sh, you can see how it runs an APK build from the command line.

I've pushed all the appropriate tags to the git repos.
Supporting creators since 2004
(When was the last time you backed up your game?)
"Do good work." - Virgil Ivan "Gus" Grissom
"Silly and fun things are important." - Elon Musk
Software > Drama • https://www.patreon.com/renpytom

User avatar
bosinpai
Regular
Posts: 100
Joined: Sat Nov 22, 2014 9:11 am
Contact:

Re: APK full rebuild from source for F-Droid

#3 Post by bosinpai » Thu Mar 26, 2015 1:10 pm

Hi,

Thanks for the pointers and for the tags - much appreciated.

The source requirement is the same as the one in Debian or Ubuntu: if you want the package to be included, you have to provide an unattended, reproducible build :)
(and of course there's a minimum of peer review so that the build can't be "cp prebuilt.apk game-release.apk" - I probably didn't express myself well :?)
Improvement over distros is that I can select dependencies versions - since I'm recompiling everything.

I'm currently debugging ./build_pgs4a.sh ("find: `-t': No such file or directory"?), hopefully this will pave the way for more renpy builds in F-Droid, including Moonlight Walks :wink:

User avatar
PyTom
Ren'Py Creator
Posts: 15893
Joined: Mon Feb 02, 2004 10:58 am
Completed: Moonlight Walks
Projects: Ren'Py
IRC Nick: renpytom
Github: renpytom
itch: renpytom
Location: Kings Park, NY
Contact:

Re: APK full rebuild from source for F-Droid

#4 Post by PyTom » Thu Mar 26, 2015 4:09 pm

You don't need build_pgs4a - you should just use build_renpy (which is called by test_renpy).

It looks like you might have an environment variable that isn't defined.
Supporting creators since 2004
(When was the last time you backed up your game?)
"Do good work." - Virgil Ivan "Gus" Grissom
"Silly and fun things are important." - Elon Musk
Software > Drama • https://www.patreon.com/renpytom

User avatar
bosinpai
Regular
Posts: 100
Joined: Sat Nov 22, 2014 9:11 am
Contact:

Re: APK full rebuild from source for F-Droid

#5 Post by bosinpai » Thu Mar 26, 2015 4:46 pm

OK, the initial SDL2 error was due to setuplib.py:library() not checking in "lib/x86_64-linux-gnu", just "lib".
(I'm not sure why build_renpy.sh tries a native compilation of "the_question" before cross-compiling to Android though)
I now get "_renpy is out of date" - progress, progress ;) [edit: cython was not installed, going on]

What is your usual build environment (os/version)?

User avatar
PyTom
Ren'Py Creator
Posts: 15893
Joined: Mon Feb 02, 2004 10:58 am
Completed: Moonlight Walks
Projects: Ren'Py
IRC Nick: renpytom
Github: renpytom
itch: renpytom
Location: Kings Park, NY
Contact:

Re: APK full rebuild from source for F-Droid

#6 Post by PyTom » Thu Mar 26, 2015 5:51 pm

Ubuntu - the latest one.

You can see the normal build environment at https://github.com/renpy/renpy .
Supporting creators since 2004
(When was the last time you backed up your game?)
"Do good work." - Virgil Ivan "Gus" Grissom
"Silly and fun things are important." - Elon Musk
Software > Drama • https://www.patreon.com/renpytom

User avatar
bosinpai
Regular
Posts: 100
Joined: Sat Nov 22, 2014 9:11 am
Contact:

Re: APK full rebuild from source for F-Droid

#7 Post by bosinpai » Fri Mar 27, 2015 10:36 am

OK, I had to do without test_renpy.sh, I couldn't find a directory layout where it would work (pre-created dist/renpy with a 'renpy' symlink to the main renpy dir?).
Instead I build rapt and copy dist/android to my $renpydir/rapt/.
I also had to patch distribute.py to avoid trying to install the basic linux/windows/mac files.

Here're my instructions to compile "The Question" for now:
EDIT: see my first post for the current documentation

Sadly the game is crashing right after the loading splashscreen :/ (note: compiling with the pre-built rapt works fine for the same device)

Code: Select all

I/python  (18045): Android path ['/data/data/org.renpy.thequestion/files', '/data/data/org.renpy.thequestion/files/lib/python27.zip', '/data/data/org.renpy.thequestion/files/lib/python2.7', '/data/data/org.renpy.thequestion/files/lib/python2.7/lib-dynload', '/data/data/org.renpy.thequestion/files/lib/python2.7/site-packages']
I/python  (18045): Android bootstrap done. __name__ is __main__
I/python  (18045): Run user program, change dir and execute main.py
I/python  (18045): /data/data/org.renpy.thequestion/files/renpy/bootstrap.py:233: UserWarning: Pygame has already been imported, import_as_pygame may not work.
I/python  (18045): Opening APK '/data/app/org.renpy.thequestion-2.apk'
I/python  (18045): Not using expansion file.
I/python  (18045): Opening APK '/data/app/org.renpy.thequestion-2.apk'
I/python  (18045): Opening APK '/data/app/org.renpy.thequestion-2.apk'
I/python  (18045): Manufacturer samsung model GT-I9000
V/OUYAF   (18045): ODK version number: 62
W/OUYAF   (18045): Not running on Ouya hardware: GT-I9000
I/python  (18045): Screen diagonal is 3.88730126323 inches.
I/python  (18045): Saving to /storage/sdcard0/Android/data/org.renpy.thequestion/files/saves
V/SDL     (18045): SDL audio: opening device
V/SDL     (18045): SDL audio: wanted stereo 16-bit 44.1kHz, 2048 frames buffer
V/SDL     (18045): SDL audio: got stereo 16-bit 44.1kHz, 3168 frames buffer
D/AudioHardware(13962): AudioHardware pcm playback is exiting standby.
V/AudioHardware(13962): open pcm_out driver
D/AudioHardware(13962): openPcmOut_l() mPcmOpenCnt: 0
V/AudioHardware(13962): openMixer_l() mMixerOpenCnt: 0
V/AudioHardware(13962): open playback normal
V/AudioHardware(13962): write() wakeup setting route HP
I/python  (18045): Logging to /storage/sdcard0/Android/data/org.renpy.thequestion/files/log.txt
I/python  (18045): Exception('Could not set video mode.',)
I/python  (18045): Full traceback:
I/python  (18045):   File "/mnt/workdir/renpy/renpy-git/renpy/bootstrap.py", line 287, in bootstrap
I/python  (18045):   File "/mnt/workdir/renpy/renpy-git/renpy/main.py", line 361, in main
I/python  (18045):   File "/mnt/workdir/renpy/renpy-git/renpy/display/core.py", line 1404, in __init__
I/python  (18045):   File "/mnt/workdir/renpy/renpy-git/renpy/display/core.py", line 1660, in set_mode
I/python  (18045): Exception: Could not set video mode.
I/python  (18045): After initialization, but before game start.
I/python  (18045): Exception: Could not set video mode.
D/dalvikvm(18045): threadid=14: thread exiting, not yet detached (count=0)
D/Sensors (  419): Smb380Sensor::~setDelay(0, 66667000)
I/WindowState(  419): WIN DEATH: Window{41e8bfb8 u0 org.renpy.thequestion/org.renpy.android.PythonSDLActivity}
I/ActivityManager(  419): Process org.renpy.thequestion (pid 18045) has died.
W/ActivityManager(  419): Force removing ActivityRecord{41bebe78 u0 org.renpy.thequestion/org.renpy.android.PythonSDLActivity}: app died, no saved state
D/Zygote  (  145): Process 18045 exited cleanly (1)
W/WindowManager(  419): Force-removing child win Window{41fcf708 u0 SurfaceView} from container Window{41e8bfb8 u0 org.renpy.thequestion/org.renpy.android.PythonSDLActivity}
W/WindowManager(  419): Failed looking up window
W/WindowManager(  419): java.lang.IllegalArgumentException: Requested window android.os.BinderProxy@41fb5a20 does not exist
W/WindowManager(  419): 	at com.android.server.wm.WindowManagerService.windowForClientLocked(WindowManagerService.java:7695)
W/WindowManager(  419): 	at com.android.server.wm.WindowManagerService.windowForClientLocked(WindowManagerService.java:7686)
W/WindowManager(  419): 	at com.android.server.wm.WindowState$DeathRecipient.binderDied(WindowState.java:1000)
W/WindowManager(  419): 	at android.os.BinderProxy.sendDeathNotice(Binder.java:470)
W/WindowManager(  419): 	at dalvik.system.NativeStart.run(Native Method)
I/WindowState(  419): WIN DEATH: null
So I mainly need to find:
  • how to compile the cython stuff without having to download lots of native libs (which I won't be able to do as a non-root build user),
  • and of course, why the game 'Could not set video mode.' :?

User avatar
bosinpai
Regular
Posts: 100
Joined: Sat Nov 22, 2014 9:11 am
Contact:

Re: APK full rebuild from source for F-Droid

#8 Post by bosinpai » Sun Mar 29, 2015 8:22 am

[updated instructions in the original post, including cython version]

Some progress: a few modules aren't compiled, in particular glblacklist.pyo and swdraw.pyo.
Consequently the graphics engine can't initialize.
I'm trying to figure out why that happened.

EDIT: work-around with

Code: Select all

(cd $WD/renpy-git && python -O -m compileall .)  # compile missing .pyo
Now we have to adapt for the F-Droid buildbot.

User avatar
bosinpai
Regular
Posts: 100
Joined: Sat Nov 22, 2014 9:11 am
Contact:

Re: APK full rebuild from source for F-Droid

#9 Post by bosinpai » Sun Mar 29, 2015 6:36 pm

So, by making all the ANGLE install conditional, skipping the install of target python&renpy.exe, and tweaking the build scripts a bit, I can rebuild a working APK. Instructions updated :)

The main issue now is running the APK builder, which requires a full, native install of Ren'Py, including Pygame/SDL2 support.
I wish it could run the APK builder without requiring all the native graphics dependencies.
Also I still haven't figured out how one can use the test_renpy.sh as-is, running scripts in the empty "dist/renpy/renpy" dir can't work without some in-between work :|

I doubt I'll have the energy to continue the investigation by myself.
EDIT: fixed APT->APK typo
Last edited by bosinpai on Mon Mar 30, 2015 4:56 am, edited 3 times in total.

User avatar
PyTom
Ren'Py Creator
Posts: 15893
Joined: Mon Feb 02, 2004 10:58 am
Completed: Moonlight Walks
Projects: Ren'Py
IRC Nick: renpytom
Github: renpytom
itch: renpytom
Location: Kings Park, NY
Contact:

Re: APK full rebuild from source for F-Droid

#10 Post by PyTom » Sun Mar 29, 2015 7:49 pm

Could you treat the output of the command line build process as source for F-droid's purposes?
Supporting creators since 2004
(When was the last time you backed up your game?)
"Do good work." - Virgil Ivan "Gus" Grissom
"Silly and fun things are important." - Elon Musk
Software > Drama • https://www.patreon.com/renpytom

User avatar
bosinpai
Regular
Posts: 100
Joined: Sat Nov 22, 2014 9:11 am
Contact:

Re: APK full rebuild from source for F-Droid

#11 Post by bosinpai » Mon Mar 30, 2015 5:01 am

Yes, basically if I copy/paste what the APK builder does in a script that I provide, although not easy to maintain, that works :)
Calling "ant" manually should be easy, I'd need to know how RAPT copies and organizes the game files, and creates the "private.mp3" archive.

User avatar
PyTom
Ren'Py Creator
Posts: 15893
Joined: Mon Feb 02, 2004 10:58 am
Completed: Moonlight Walks
Projects: Ren'Py
IRC Nick: renpytom
Github: renpytom
itch: renpytom
Location: Kings Park, NY
Contact:

Re: APK full rebuild from source for F-Droid

#12 Post by PyTom » Mon Mar 30, 2015 9:17 am

You shouldn't need to call ant manually. Once you have the game files laid out properly, you should be able to call android.py build <path-to-files> and it will manage the latter half of the process.
Supporting creators since 2004
(When was the last time you backed up your game?)
"Do good work." - Virgil Ivan "Gus" Grissom
"Silly and fun things are important." - Elon Musk
Software > Drama • https://www.patreon.com/renpytom

User avatar
bosinpai
Regular
Posts: 100
Joined: Sat Nov 22, 2014 9:11 am
Contact:

Re: APK full rebuild from source for F-Droid

#13 Post by bosinpai » Mon Mar 30, 2015 9:35 am

PyTom wrote:Once you have the game files laid out properly,
Yeah, that's precisely what's difficult to do without the renpy launcher.

Post Reply

Who is online

Users browsing this forum: No registered users