We successfully transferred faces and achieved reasonable results. Areas to improve upon include better blending between faces with different skin colors and tracking features that disappear mid-frame (e.g. the lip)
Workflow of our algorithm (swapping between single frames)
We successfully transferred faces and achieved reasonable results. Areas to improve upon include better blending between faces with different skin colors and tracking features that disappear mid-frame (e.g. the lip)
Face Swapping
My final project for a course in Computer Vision (CIS 581) involved detecting and swapping faces between videos (similar to the popular SnapChat feature).
Algorithm
After choosing an image to extract a face from ("source"), we use Contrast-Limited Adaptive Histogram Equalization in order (CLAHE) to make facial features more distinguishable in the image. We detected facial landmarks on the "source" image using the open source Dlib Python library. Using these landmarks, we perform a Delaunay Triangulation to warp the features from the source image in line with corresponding features in the target image.
We then form a convex hull mask, using which we extract pixels from the source and swap them with the corresponding pixels in the target image. Once the pixels are replaced, we use seamless cloning (gradient domain) blending to make the transition of colors more visually realistic. Finally, optical flow tracking is used to track the movement of the face over the course of the video.