Lip synchronization with a sound file

[Encritary]

Hello! I found and tried some Blink and Lip Flap scripts, including the one with playing sound detection. But lip animation still looks horribly unnatural. I want to make lip synchronization with the current sound file playing in the voice channel, just like in professional-made visual novels, for ex. SciADVs. Is it possible for Ren'Py to do so? Even if you have to do some hard-code directly with Python.

[wyverngem]

How realistic do you want the mouth animations? You could set up it's own ATL for a layeredimage and find a way to animate it with code. However, I think it would mean you going line by line to change the animations.

[Remix]

It is certainly do-able in Ren'Py, though you will have to be reasonably familiar with Python and get to know how things such as auto-voice and DynamicDisplayables work. I would suggest, if you do try this as a starter project, look at something like Rhubarb LipSync to generate files containing the time=>lip_shape info, read that in and use a Dynamic Displayable to output just the characters mouth. A DD will/can update its shown image dynamically at every frame, so you'd want to just map the current position of the voice audio and resolve which mouth shape to show.


(note: audio is rather low volume, oops)

https://github.com/DanielSWolf/rhubarb- ... c/releases

I've got a long day at work tomorrow so need sleep now. Might be able to help more later in the week.

[Andredron]

Hello! I found and tried some Blink and Lip Flap scripts, including the one with playing sound detection. But lip animation still looks horribly unnatural. I want to make lip synchronization with the current sound file playing in the voice channel, just like in professional-made visual novels, for ex. SciADVs. Is it possible for Ren'Py to do so? Even if you have to do some hard-code directly with Python.

viewtopic.php?f=8&t=4010&start=15