Lip synchronization with a sound file

Discuss how to use the Ren'Py engine to create visual novels and story-based games. New releases are announced in this section.
Forum rules
This is the right place for Ren'Py help. Please ask one question per thread, use a descriptive subject like 'NotFound error in option.rpy' , and include all the relevant information - especially any relevant code and traceback messages. Use the code tag to format scripts.
Post Reply
Message
Author
Encritary
Newbie
Posts: 2
Joined: Sun Mar 10, 2019 11:51 am
Github: encritary
Contact:

Lip synchronization with a sound file

#1 Post by Encritary » Sun Mar 10, 2019 11:57 am

Hello! I found and tried some Blink and Lip Flap scripts, including the one with playing sound detection. But lip animation still looks horribly unnatural. I want to make lip synchronization with the current sound file playing in the voice channel, just like in professional-made visual novels, for ex. SciADVs. Is it possible for Ren'Py to do so? Even if you have to do some hard-code directly with Python.

User avatar
wyverngem
Miko-Class Veteran
Posts: 594
Joined: Mon Oct 03, 2011 7:27 pm
Completed: Simple as Snow, Lady Luck's Due,
Projects: Aether Skies, Of the Waterfall
Tumblr: casting-dreams
itch: castingdreams
Location: USA
Contact:

Re: Lip synchronization with a sound file

#2 Post by wyverngem » Sun Mar 10, 2019 4:33 pm

How realistic do you want the mouth animations? You could set up it's own ATL for a layeredimage and find a way to animate it with code. However, I think it would mean you going line by line to change the animations.

User avatar
Remix
Eileen-Class Veteran
Posts: 1315
Joined: Tue May 30, 2017 6:10 am
Completed: None... yet (as I'm still looking for an artist)
Projects: An un-named anime based trainer game
Contact:

Re: Lip synchronization with a sound file

#3 Post by Remix » Sun Mar 10, 2019 6:16 pm

It is certainly do-able in Ren'Py, though you will have to be reasonably familiar with Python and get to know how things such as auto-voice and DynamicDisplayables work. I would suggest, if you do try this as a starter project, look at something like Rhubarb LipSync to generate files containing the time=>lip_shape info, read that in and use a Dynamic Displayable to output just the characters mouth. A DD will/can update its shown image dynamically at every frame, so you'd want to just map the current position of the voice audio and resolve which mouth shape to show.


(note: audio is rather low volume, oops)

https://github.com/DanielSWolf/rhubarb- ... c/releases

I've got a long day at work tomorrow so need sleep now. Might be able to help more later in the week.
Mad Scientist Todo List:
  • Finish Improved Event Handler
  • Implement DragonBones animation as a Creator Defined Container
  • Develop Cartoon Speech Bubble dialogue
  • Finish Bitmask collision and rebound vector system
  • Develop time based building mechanic
  • Others
  • *Find a superb artist and actually write a game*

User avatar
Andredron
Veteran
Posts: 347
Joined: Thu Dec 28, 2017 2:37 pm
Location: Russia
Contact:

Re: Lip synchronization with a sound file

#4 Post by Andredron » Thu Mar 14, 2019 8:33 am

Encritary wrote:
Sun Mar 10, 2019 11:57 am
Hello! I found and tried some Blink and Lip Flap scripts, including the one with playing sound detection. But lip animation still looks horribly unnatural. I want to make lip synchronization with the current sound file playing in the voice channel, just like in professional-made visual novels, for ex. SciADVs. Is it possible for Ren'Py to do so? Even if you have to do some hard-code directly with Python.
viewtopic.php?f=8&t=4010&start=15
I know, I'm writing terribly in English.

I'm writing a Renpy textbook (in Russian). https://yadi.sk/d/ZX_DonP63USRru Update 22.06.18

Honest Critique

Post Reply

Who is online

Users browsing this forum: amekoi