Research

Pylon Deformer and Pipeline

Pylon is a pipeline and plugin that allows for real-time deformations in games and Unreal Engine. Utilizing VFX-style simulations to push deformation back into models and rigs for real-time performance.

Creating a better deformer as well as organizing and simplifying the workflow needed to achieve this by using my own pipeline, Pylon.

Auto Muscle Fibre Firing

Generally in creature work in the film industry, muscles are fired or activated based on either hand-animated values, or automated by some calculation of a vector changing length between two points, a foot and a hip, a wrist and a shoulder, and so on. This can definitely give the illusion of muscles firing, but it isn’t really accurate. So I wanted to explore other ways to do this. Below is what I ended up with after exploring various options.

A Simple CNN from Labelled Photos

I was originally playing around with using a simple CNN based on a bunch of images I took of my own arm, which I had manually labelled with muscle activation values for some of the basic muscles:

• Biceps (short head, long head)
• Triceps (long head, lateral head)
• Deltoids (front, side, rear)
• Forearm muscles (F1 to F16)
• Wrist and finger muscles (W1 to W12)

Then I created a simple CNN using PyTorch, with the labelled arm images as the inputs and the outputs being the multiple muscle activation values.

From there I wrote a script to load the saved model, accept a new single image or a batch, so a rendered 3D playblast of the frames for a shot, and predict the muscle activations from that. I also explored using a 3D CNN to try and process sequences for better temporal prediction.

To get this working better it would need far more training data, and overall it is a bit inaccurate. But it was something I wanted to try for fun. There are many flaws and limited accuracy, but it was something I could put together quickly, since I could just use photographs of my own arm and create the data myself.

EMG Data and Neural Networks

I started looking at how muscle firing can be measured more accurately, and found a range of EMG data, in particular a gesture recognition and biometrics electromyogram dataset, the GrabMyo database. Details of the experimental protocol, participants and electrode locations are documented in the files that ship with the dataset.

I used some of this data to try and create a neural network I could run other poses through to automatically generate the fibre fire values. I built my own matching poses in 3D based on the ones the participant made in the study, exported the FBX Maya poses to .npy, used the pose and session data to create a JSON containing the EMG keyframes, with input and target .npy files holding the X and Y channels mapped from the EMG data, and then trained the network, training the poses to EMG.

Then I can run inference through different poses, taking that new data and generating new muscle fire values for that motion. The last step re-maps that fibre fire data back to the zFiber muscle activations I made in my muscle sim setup, matching the original capture points from the EMG, which lets me control the excitation of the muscle.

This was great, and digging through all of it got me more comfortable with larger datasets and with setting up a network to give me… something. Part of the problem was that there was no visual data on the participants’ hand poses specifically (mocap, filmed footage), so after normalizing the data and playing with it, it is hard to comfortably dial in, as I do not know the type of arms, hands or skin involved. There was per-user information, but it is still a bit hit and miss for my use case, similar to the problems with my CNN and 3D CNN.

OpenSim and Biomechanics Data

That got me digging further and I ended up looking into OpenSim, and how the medical and veterinary peeps use it to accurately calculate a whole range of things. The datasets vary, but there is so much out there, and a lot of it includes mocap/motion data in some form, which makes it far more usable.

Have a look at the EMG project on SimTK.

This was really fun to dig into, and I went off on plenty of tangents testing horse muscle firing and other animals, since there was a wealth of information there too.

Into Unreal Engine

In the end I used a combination of EMG data and photogrammetry to create the data to train my realtime deformer, which I added to Unreal Engine.

Below are some tests and WIPs from the process of adding it to UE.

Unreal Tests and Ideas

Short experiments and prototypes.