Pumpkin Lip Sync
Introduction: Pumpkin Lip Sync
Make your own pumpkin jack o' lantern lip sync videos.
There are plenty of sources to purchase premade jack o' lantern animations: atmosfx.com, thesingingpumpkin.com. There is even a pumpkin with an integrated short throw projector, Jabberin' Jack. If you want to make a custom or DIY animation there is commercial software you can utilize such as Adobe After Effects. But now you can (with less skill) make your own using free and open source software.
I wrote this application in HTML and JavaScript. You can customize the images as you (and your talent) see fit. 141
It utilizes an open source software package for analyzing the audio to generate an XML file of "mouth shapes", that the HTML can than load to play the audio and move the mouth accordingly. The HTML application lets you adjust/override the mouth shapes, and make the eyes move; so you can fine tune the animation.
You can either play the animation directly from you computer projected onto a pumpkin, or use a video capture tool to convert into a video that can be loaded onto your projector.
I also provided a face for animating a snowman, in addition to a jack o' lantern. Why is there a bird (cockatiel) face? My pal Launchpad has dreams of being an opera singer. (actually early test, and it amused me).
Supplies
* Rhubarb Lip Sync
* https://github.com/DanielSWolf/rhubarb-lip-sync
* Pumpkin Lip Sync HTML application (and a web browser to run it in)
* https://www.dropbox.com/s/85low7msao1pas1/lipsync.zip?dl=0
* Screen Capture utility (optional).
* I used ScreenCastify, a plug-in for Chrome. It records 10 minutes for free.
* www.screencastify.com
* MP3 to WAV converter (optional) (Rhubarb only supports WAVE (.wav) and Ogg Vorbis (.ogg)).
* https://convertio.co/
* MP3 splitter for seperating music and vocal tracks (optional)
* If lip syncing a song, the results are much better if removing the music before feeding into Rhubarb.
* https://www.edityouraudio.com/acapella-generator (didn't work for me)
* https://www.acapella-extractor.com/ (limit of two per day)
* https://www.lalal.ai/
* Foam or real pumpkin (or 3D printed pumpkin)
* Projector
Step 1: Record Your Audio for the Animation
You can use the audio recorder built into Windows to prepare some jokes or stories for your pumpkin to tell.
Or you can find a song for your pumpkin to sing.
Convert the audio to a WAV file.
If you're using a song, use a tool such as Acapella-Extractor.com to separate the music and vocal audio tracks.
Extract vocals from music (for better lip sync processing)
https://www.edityouraudio.com/acapella-generator (didn't work for me)
https://www.acapella-extractor.com/ (limit of two per day)
https://www.lalal.ai/
You may still need to remove/flatten sections of audio that were missed. You'll need to download a sound editor for that.
Convert both copies of the MP3/MP4A to WAV
https://convertio.co/
Step 2: Run Audio Through Rhubarb
Using the command line have Rhubarb analyze the vocal audio track, and generate a JSON file of the mouth shapes.
Install Rhubarb Lip Sync
https://github.com/DanielSWolf/rhubarb-lip-sync
Process WAV file (w/ vocals only) creating JSON output (used for English recordings)
rhubarb -o [target filename].json -f json [vocal audio filename].wav
rhubarb -o barber.json -f json baber_largo_farctum.wav
Process WAV file (w/ vocals only) creating JSON output, using phonetic interpretation (use for non-English recordings)
rhubarb -r phonetic -o [target filename].json -f json [vocal audio filename].wav
rhubarb -r phonetic -o Bella-Ciao-phonetic.json -f json La-Casa-De-Papel-Bella-Ciao-_Money-Heist_.wav
Step 3: Modify the JSON File
Modify the JSON data.metadata. 141