Real-Time Photo Style Transfer

Hanli Zhao1
1 2

Xiaogang Jin1

Jianbing Shen2

Feifei Wei1

State Key Lab of CAD & CG, Zhejiang University, China School of Computer Science & Technology, Beijing Institute of Technology, China Corresponding author. Email: jin@cad.zju.edu.cn Abstract
of the luma, pure yellow-blue, and pure red-green channels. To address the problem, Bratkova et al. [2] introduced a new oRGB color space based on opponent color theory. Thus, the statistical properties of the channels can be transferred in a meaningful space in terms of natural primaries and secondaries. However, both methods are implemented on the CPU and the global statistical analysis is traditionally viewed as a highly sequential step. The main limitation lies in the high computational cost as transferring an images color requires several seconds. It is even worse when processing an online video. On the other hand, in addition to the color, the luminance itself carries a lot of visual information. Matching histogram is the traditional solution to transfer an intensity distribution. We observed that matching the pixel histogram plays an important role in transferring the tonal look of an image, which cannot be achieved with color transfer alone. In this paper, we present a novel approach for real-time photo style transfer. First of all, the input image is converted from RGB color space to Bratkova et al.s oRGB [2] color space. Then we use a similar statistical analysis proposed by Reinhard et al. [8] to impose an images color characteristics on another. In addition, we perform the histogram matching to achieve better photographic look. To achieve real-time performance, we employ the highly parallel CUDA computing environment to calculate the means and standard deviations, as well as the histogram accumulation. Note that the proposed algorithm outperforms

This paper presents a novel approach for realtime photo style transfer. The automatic image manipulation technique is performed in the oRGB color space, which is a new color model based on the psychologically opponent color theory. We transfer color from an appropriate source image to the target image using a simple statistical analysis. In addition, we match the global luminance histogram to achieve better photographic look. Note that the whole pipeline is highly parallel, enabling a GPU-based real-time implementation. Several experimental results are shown to demonstrate the effectiveness and efciency of the proposed method.

1. Introduction
The color and luminance are two basic elements of an image. Probably the most common task in image processing is to alter an images color or intensity. Many users seek to obtain a certain look for their pictures to convey a mood or an sthetic, which has led researchers to investigate new image processing techniques. The pioneering work of color transfer was done by Reinhard et al. [8]. The method imposes one images color characteristics on another using a simple statistical analysis of means and standard deviations. They demonstrated that the l color space provides excellent results when compared to the RGB color space. Although l works well in many situations, it lacks independent manipulation

_____________________________

978-1-4244-3701-6/09/$25.00 2009 IEEE

6RXUFH ,PDJH

,QSXW

R5*% &RORU 6SDFH

+LVWRJUDP 0DWFKLQJ

$EVWUDFWHG

&RORU 7UDQVIHU

Figure 1. Workow overview. state-of-the-art methods with both better visual results and higher processing efciency. As a result, our new approach enables real-time online video processing. In this paper, we would like to develop a photo style transfer system that is fully automatic and real-time.

2. Our Approach 1.1. Related Work

Many style transfer techniques have been developed in computer graphics and imaging communities. Reinhard et al. [8] described a method for a general form of color correction that borrows one images color characteristics from another in the l color space. Bratkova et al.[2] improved the color transfer results by introducing the oRGB color space. Welsh et al. [11] introduced a general technique for colorizing greyscale images by transferring color between a source, color image and a destination, greyscale image. Tai et al. [9] and Chang et al. [3] presented a probabilistic approach to conduct local color transfer, while Wen et al. [12] provided a direct indication mechanism with a stroke-based user interface. Hertzmann et al. [6] instead explored the style transfer for the textural aspects of non-photorealistic media. These approaches, however, are implemented on the CPU and thus cannot process online images and videos. The basic workow of our framework is shown in Figure 1. We rst transform the input image from RGB color space to oRGB color space. Then the statistical color correction technique is performed on the two chroma channels. Next we apply the histogram matching on the luminance channel. Optionally, we abstract the style transferred image to further improve the visual perception. At last, the result image is produced by transform back to RGB color space. When transferring a source images style to a video, since the source image is unchanged, we can compute its statistical information only once to reduce the processing time.

2.1. oRGB Color Space

The oRGB model, which is a new color space introduced by Bratkova et al.[2], is an invertible transform from RGB. The primaries of this model

are based on the three fundamental psychological opponent axes (white-black, red-green, and yellow-blue). The nonlinearity of these three channels relative to intensity uses the nonlinear gamma encoding of the native RGB space. There are two steps to perform the transform. In step 1, oRGB transforms the RGB cube into the LCC color space via a linear transform: L 0.2990 0.5870 0.1140 R C1 = 0.5000 0.5000 1.000 G 0.8660 0.8660 0.0000 B C2 (1) and its inverse is:

oRGB space to transfer between images. In this subsection, we describe how to transfer the redgreen and yellow-blue channels using the mean and standard deviations along each of the two axes. We employ a statistical analysis similar to Reinhard et al. [8]. We compute these global measures for both the source and target images. Then we subtract the mean from the target data points: t Cyb Cyb Cyb = (6) t Crg Crg Crg Next we scale the data points comprising the synthetic image by factors determined by the respective standard deviations: Cyb Crg =
s t (yb /yb )Cyb s t (rg /rg )Crg

R 1.0000 0.1140 0.7436 L G = 1.0000 0.1140 0.4111 C1 C2 B 1.0000 0.8660 0.1663 (2) In step 2, we perform the nonuniform rotation around luma axis L from LCC (L , C1 , C2 ) to oRGB (L, Cyb , Crg ). The luminance L equals to L . For points above the yellow-blue axis, the angle in the linear chroma plane is = atan2(C2 , C1 ). The new angle in oRGB space o is:

(7)

After this transformation, the resulting data points have standard deviations that conform to the source photograph. Next, instead of adding the averages that we previously subtracted, we add the averages computed for the source photograph s s [Cyb , Crg ].

(3/2) if < /3; o () = /2 + (3/4)( /3) if /3. Matching histograms is the traditional solution (3) to transferring an intensity distribution [4]. Given To compute the point (Cyb , Crg ) in oRGB we an image I with histogram ht and a source hissimply rotate the (C1 , C2 ) point: togram hs , we seek a function f such that applying f to each pixel of I results in an image with Cyb C1 histogram ht . To build f , we use the cumulative = R(o ) (4) Crg C2 x histograms ct and cs dened by c(x) = h. It can be shown that cs (I) produces an image with The inverse transformation angle from oRGB to a uniform histogram. Thus, we dene: LCC is: (8) f (x) = c1 [cs (x)] (2/3)o if o < /2; t (o ) = /3 + (4/3)(o /2) if o /2. and f (I) generates an image with the histogram (5) ht . We observed that matching the pixel histogram is a key tool to control the tonal look of an im2.2. Transferring Color Style age, which cannot be achieved with color transfer alone. The goal of our work is to make a synthetic Instead of transferring luminance distribution image take on another images look and feel. with the simple statistical analysis as used by More formally this means that we would like Reinhard et al. [8], we employ the histogram some aspects of the distribution of data points in matching to obtain better photographic look.

2.3. Matching Luminance Histogram

2.4. Highly Parallel Implementation

The proposed algorithms are embarrassingly parallel. Each pixel in the image seems to be processed individually. However, the statistical measures must be computed in a global fashion. Moreover, the nave implementations for his togram accumulation and matching are also quite time-consuming. To address the bottleneck, we propose a highly parallel solution, enabling a realtime GPU-based implementation. Recently, NVIDIA Corporation released CUDA [7], a relatively new hardware and software architecture for managing the GPU as a data parallel computing device. CUDA supports highly efcient gathering and scattering operations. In addition, the on-chip shared memory accelerates the communication among threads within a thread block. In this work, we employ CUDA for efcient photo style transfer and Direct3D 10 API [1] for rendering. The computation of the means and standard deviations is actually a global gathering operation. Harris et al. [5] developed a CUDA implementation for parallel prex sums. First the input data are divided across multiple thread blocks and the partial sum tree results are fast computed by taking advantage of the shared memory. Then the results are used as input to a single thread block to get the nal value. In our work, we do some optimizations to further improve the performance. We observed that up to 512 threads per block are supported on current CUDA platforms. Thus we explicitly unroll the loops when traversing the shared memory. In addition, we write the perblock result only once. Scheuermann and Hensley [10] presented an efcient algorithm to compute image histograms using additive alpha blending. We can also implement the algorithm using CUDA, because CUDA supports efcient atomicAdd function for integers [7]. When performing the histogram matching, we load the histogram values into the shared memory to improve the traversal efciency.

3. Results and Applications

We have developed a new automatic real-time photo style transfer system for images and videos based on Direct3D 10 and CUDA, and tested it on some scenes in order to evaluate its efciency. All tests were conducted on a PC with a 2.40 GHz Intel Core 2 Quad CPU, 2GB main memory, an NVIDIA Geforce 8800 GTS GPU, 512 MB graphics memory, and Windows Vista operating system.
Image size Performance 0.5 7 1.0 13 1.5 22 2.0 25 3.0 36

Table 1. Running times (ms) of our photo style transfer system. The image size is in mega-pixels.

We show the running timings in Table 1. We set the size of histogram bins as 256, which is just the maximum contrast for typical RGB-format pictures. In our current work, the video decoding is performed on the CPU in a separate thread that runs in parallel with the GPU. From the table we can see that we are able to perform the style transfer at 27 Hz even for images with 3.0 megapixels, satisfying sufcient real-time requirement on modern PC monitors. Figure 2 shows the comparison of the style transfer result between our new approach and Reinhard et al.s method. Although Reinhard et al.s l works well in many situations, oRGB provides features lacking in l. For example, color transfer in oRGB allows for independent manipulation of the luma, pure yellow-blue, and pure red-green channels. Thus, statistical properties of the channels can be transferred in a meaningful space in terms of natural primaries and secondaries. Orthogonality is not the full story for color transfer. Although l produces decorrelated axes using principle component analysis, oRGB uses opponent-color perceptual axes for meaningful computational applications. As Figure 2 shows, our new approach produces better result than Reinhard et als method.

Other style transfer effects automatically generated in our system are shown in Figure 3. We can see how yellow-blue and red-green channels are transferred, and how luminance histograms are matched to obtain better photographic look. Moreover, the accompanying video demonstrates that our photo style transfer system can generate vivid visual perception. The video sampling rate is 30 Hz. Here we show that our style transfer component can be used as a preprocessing step for real-time video and image abstraction. After transferring photo style from a source image, the target image can then be abstracted for better illustration. In this paper, we employ Zhao et als feature-aware abstraction framework [13] to get the abstracted results. The method employs the bilateral grid to simplify regions of low contrast and use a feature ow-guided anisotropic edge detection lter to enhance regions of high contrast. As shown in Figures 2e, 3d, 3i, and 3h, our method can be an interesting complement for real-time video abstraction.

References
[1] Blythe, D.: The Direct3D 10 system. Proc. ACM SIGGRAPH 06, pp. 724734. ACM, New York (2006) [2] Bratkova M., Boulos S., Shirley P.: oRGB: a practical opponent color space for computer graphics. IEEE Computer Graphics and Applications 29, 1 (2009), pp. 186196. [3] Chang Y., Saito S., Nakajima M.: Example-based color transformation of image and video using basic color categories. IEEE Transactions on Image Processing 16, 2 (2007), pp. 329336. [4] Gonzales R. C., Woods, R. E.: Digital image processing. Prentice Hall (2002) [5] Harris M., Sengupta S., Owens J. D.: Parallel prex sum (scan) with cuda. In GPU Gems 3, Nguyen H., (Ed.). Addison Wesley (2007) [6] Hertzmann A., Jacobs C. E., Oliver N., Curless R., Salesin D. H.: Image analogies. Proceedings of ACM SIGGRAPH 01, pp. 327340. ACM, New York (2001) [7] NVIDIA Corp.: NVIDIA CUDA programming guide. http://developr.nvidia.com/cuda, version 2.1 (2008) [8] Reinhard E., Ashikhmin M., Gooch B., Shirley P.: Color transfer between images. IEEE Computer Graphics and Applications 21, 5 (2001), pp. 28. [9] Tai Y. W., Jia J., Tang C. K.: Local color transfer via probabilistic segmentation by expectation maximization. Proceedings of Computer Vision and Pattern Recognition, pp. 747754. (2005) [10] Scheuermann T., Hensley J.: Efcient Histogram Generation Using Scattering on GPUs. Proceedings of ACM Interactive 3D Graphics and Games 07, pp. 3337. ACM, New York (2007) [11] Weilsh T., Ashikhmin M., Mueller K.: Transferring color to greyscale images. ACM Transactions on Graphics 21, 3 (2002), pp. 277280. [12] Wen C. L., Hsieh C. H., Chen B. Y., Ouhyoung M.: Example-based multiple local color transfer by strokes. Computer Graphics Forum 27, 7 (2008), pp. 17651772. [13] Zhao H., Jin X., Shen J., Mao X., Feng J.: Real-time feature-aware video abstraction. Visual Computer 24, 7 (2008), 727734.

4. Conclusions
In this paper, we have presented a novel photo style transfer system that automatically transfers a source images style to a target image or video. The algorithms are highly parallel, allowing a real-time implementation on modern GPUs. Experimental results have shown that our proposed approach produces better results than state-of-theart methods. Additionally, the experimental results demonstrate both the feasibility and efciency of our proposed algorithms.

Acknowledgments
his work was supported by the National Natural Science Foundation of China (Grant nos. 60533080 and 60833007) and the Key Technology R&D Program (Grant no. 2007BAH11B03).

(a) Source image

(b) Target image

(c) Color transferred

(d) Histogram matched

(e) Our abstracted effect

(f) Reinhard et al.s result

Figure 2. Comparison of our new approach with Reinhard et al.s method.

(a) Source image

(b) Target image

(c) Style transferred from image a

(d) Our abstracted result

(e) Source image

(f) Target image

(g) Style transferred from image d

(h) Style transferred from the image in Figure 2a

(i) Abstracted from image g

(j) Abstracted from image h

Figure 3. Various interesting photo style transfer results using our approach.