High-quality, cost-effective facial motion capture pipeline with 3D regression

We present our improved marker-based facial motion capture pipeline that leverages on 3D regression from head-mounted camera (HMC) images to speed up and reduce the cost of high quality 3D marker tracking. We use machine learning to boost productivity by training regressors on traditionally tracked performances and applying those models to the remaining performances. Our specialized regressor for HMC marker-based tracking shows improvements in quality and robustness for marker tracks. The regressor results are automatically refined by a simple blob detection tool and then imported back into the tracking tool such that manual correction can be applied as needed and subsequently included as additional training data. This iterative approach reduces 70% the amount of artist time required for traditional tracking methods and does not add much setup time nor planning as alternative techniques.