Face Triangulation for Graphic Design

Nowadays low poly has emerged as a popular element in graphic design, creating sculpture/crystal-like effects. Various movie posters utilize this technique to highlight characteristics of the main subject, e.g. the Star Wars poster shown below. And multiple mobile apps aim to make this visual effect more approachable for common users.

Image Credit: Star Wars low poly portraits, designed by Vladan Filipovic

To convert a posed photo to low poly style, designers employ different tools and emphasize different aspects. Among all these variants, a typical production process involves three steps: 1. place vertices (usually 500~5000) on suitable locations of image; 2. generate mesh from vertices using Delaunay triangulation; 3. fuse colors inside each triangle-coverage area.

The last two steps are quite straightforward and there are several off-the-shelf tools for them. What takes most effort and exhibits unique aesthetic taste is the first step, i.e. choosing the right locations on image to place vertices. So in the post, we attempt to automate this vertices selection step by referring to computer vision approaches, especially in face triangulation domain.

Generally two principles define a good set of vertices: 1. vertices should be distributed more or less uniformly, so that the shear of each triangle generated wouldn't be dramatic; 2. On the boundaries of semantic parts or textured details, vertices should be more densely, such that unions of triangles would correspond to semantic parts. In this sense, the problem has been cast onto seeking a proper probability density map for sampling.

Here we advocate to combine face alignment and boundary detection to produce a desired density map. Dense facial landmarks will place more emphasize on semantic facial parts (e.g. eyebrows, eyes, nose, mouth etc.) and edge maps will preserve object contours and fine details in image. My lovely Alice and two dear professors are taken as models and the final effect is illustrated as follows.

Image Credit: Prof. Xiaoou Tang & Prof. Xiaogang Wang

The current result seems surreal but nice. Future improvements could be made on two directions: 1. Devise post-processing method to re-split and re-merge triangles according to established scale distribution; 2. Form an end-to-end system for learning point correspondences, just like Pointer Networks.

Extrapolating Paintings by PatchMatch

PatchMatch is a popular image editing technique for establishing dense correspondence in or across images. A group of Cambridge students utilized this technique to extrapolate classic paintings (e.g. Van Gogh's 'Starry Night') that went beyond frame constraints and achieved fantastic results.

The key ingredients of PatchMatch are two insights on natural image statistics, especially from the patch offsets space: 1. There are abundant similar patches inside natural image (Non-local Means also shares this spirit); so large number of random sampling would yield good initialization. 2. Neighboring patches often have similar offset vectors; this observation enables propagation of offset vectors among neighboring patches that leads to an efficient iteration solution.

I have also tried this useful algorithm on one of Jimmy Liao's hand drawings and a Chinese ancient painting. The extrapolation effects are illustrated as follows:

Both results are visually plausible overall. By inspecting some local regions and details, we further find that: 1. This category of methods (like PatchMatch) work well on smooth regions, repetitive patterns and could also maintain local semantic coherence. 2. Global semantic coherence still can't be handled, which leaves room for the interdisciplinary research between low-level image processing and high-level visual reasoning.   

The Secret of Bokeh

Bokeh, the technique of deliberately creating out-of-focus area in photos, has been widely used by photographers for emphasizing semantic subjects. These subjects could be either foreground or background of photos. An example of portrait bokeh is shown below:

Photograph by Di Liu

The effect of bokeh is usually introduced by particular lens in SLR cameras. However, camera in mobile devices has become more and more ubiquitous and also a common choice to record daily life nowadays. So the question arises that how can we design algorithms to generate the effect of bokeh in well-focused photo shot by mobile devices.

In fact, this functionality (creating semantically out-of-focus photo from well-focused photo) is a well-studied topic in computational photography. And a typical algorithm comprises two stages: depth map estimation and selective blurring.

We would like to obtain depth map of the underlying photo in the first place to distinguish between foreground and background. But estimating depth map from a single image has long been known as a difficult task. A practical solution is to leverage information in image sequence (burst mode in mobile devices) so that a multi-view stereo could be formed. Google's Lens Blur adopts this kind of practice and they had a research blog to elaborate detailed techniques. BTW, burst images could also be utilized for denoising. After that, we can perform adaptive per-pixel blurring according to the computed depth map. Or more advanced technique could be adopted to enable motion blur rendering. To make the final result more seamless and natural, the above rendering process could be done in a multi-scale manner, e.g. local Laplace pyramids. The same pipeline could also be applied to generate 3D effect image from burst images, as demonstrated in this website.