Years ago, I was blown away by an emerging new way to use an iPhone for facial motion capture. Kite and Lightning posted a video in which he was doing real time motion capture using nothing but his iPhone. No markers. No fancy cameras or elaborate setup or post processing. Just his face and his phone. At the time, it required a bit of hacking to make it work. But now, two years later, the process has become a lot simpler. So simple, in fact, that now even I can do it. While I still have a lot of work to go, I wanted to share my progress and show where I'm at now getting this amazing new ability in place to use with the Thera characters.
For most people who see this, one of their first questions is, "Do I need to use an iPhone?" The answer is, "yes". The iPhone X (and later) has a special camera that allows more accurate facial capture. Apple's TrueDepth camera system is what makes it special, and why you can't use just any camera. (There are ways to capture facial animation with a regular camera, but they aren't as accurate.) There are several applications that can be used to leverage this camera and the facial data it can capture. For my tests, I have been using Unreal's free Live Link Face. There were several great resources I used to help get this set up. Check out these 2 videos:
Live Link Face Remapping Tutorial | Unreal Engine
iPhone Facial Capture with Unreal Engine | Unreal Fest Online 2020
One thing that wasn't covered was how to get the head movement mapped to the bones. I had to do some digging into Unreal's docs to find the answers.
It was still a bit confusing, and took a couple tries to get the nodes wired up properly. I've attached a screen grab of what my final setup looked like.
While this still needs a bit more work, a quick initial setup has yielded some pretty promising results. The video below was a quick and dirty test I did just to see how it was coming together. There is a bit of a lag in the capture, so the audio was slightly offset in the edit to compensate for the lag. Aside from that, this is the raw test clip. As mentioned, I still have a bit of work to do. Once everything is set up and running smoothly, I'll post another blog with the full breakdown of the process and setup I used.