深圳大学学报理工版

图像风格迁移是对给定的输入图像,采用计算机手段自动将任意其他图像的风格迁移到输入图像中,使两者风格完全相同,但保持原图像内容不变.本研究通过实验发现,当这种风格迁移用到人脸图像时,由于嘴唇、眼睛等部位对空间位置差异比较敏感,容易导致风格迁移效果失真.基于DPST(deep photo style transfer)算法,提出对空间差异鲁棒的人脸图像风格迁移的改进算法,通过将输入图像和风格图像各部分语义的分割内容在深度卷积神经网络中特征图通道间的归一化互相关最大化,对风格图像进行对应的仿射变换,每个分割区域都以对应的空间变换后的样式图像作为样式参考,有效减轻了输入图像和风格图像各部分语义分割内容在空间上存在较大差异时产生的重影.选取空间差异较大的输入图像与风格图像,对新算法和原DPST算法以及其他基准算法进行对比实验,结果表明,当两张图像各语义分割内容在空间位置相差较大时,新算法仍能产生较好的效果,鲁棒性更佳.

For an input image, the image style transfer is to change the style of the input image accordingly without modifying its original content. Our empirical studies discover that, when the existing style transfer techniques are applied to headshot portraits, the noticeable distortions could be incurred due to the fact that the relative location of those facial components, such as lips, noses, and eyes, are sensitive to spatial differences. Based on the DPST(deep photo style transfer)method, a spatially-robust image style transfer algorithm for headshot portraits is proposed in this paper to improve the robustness of image style transfer. The proposed algorithm provides an improved solution for the problem that the transferred content could incur ghost shadows when segmented regions between the input image and the style image are significantly different. By maximizing the normalized cross-correlation of the channels of feature maps of corresponding segmented regions of the input image and the style image in a pre-trained convolutional neural network, we propose to apply affine transformations to the style image before it is used for the style transfer. Each segmented region takes the corresponding spatially transformed style image as style reference, making it adaptive to the spatial difference between the input and the style image and the incurred ghost shadow can be minimized or eliminated. The experimental results show that our proposed algorithm can obtain good results even when the semantic segmentation between two images have the large spatial differences. The new algorithm also displays the better robustness than the DPST algorithm and other benchmark.

引言
1 具有真实细节的图像风格迁移
2 对空间差异鲁棒的人脸图像风格迁
3 实验结果及分析
4 结语

图1 空间变换算法示意<br/>Fig.1 Spatial transformation algorithm

图1 空间变换算法示意
Fig.1 Spatial transformation algorithm

图2 DPST算法、SHIH算法与本研究算法的实验结果对比<br/>Fig.2 Comparative illustration of experimental results

图2 DPST算法、SHIH算法与本研究算法的实验结果对比
Fig.2 Comparative illustration of experimental results

[1] REINHARD E, ASHIKHMIN M, GOOCH B, et al. Color transfer between images[J]. IEEE Computer Graphics and Applications, 2002, 21(5): 34- 41.
[2] SUNKAVALLI K, JOHNSON M K, MATUSIK W, et al. Multi-scale image harmonization[J]. ACM Transactions on Graphics, 2010, 29(4): 125.
[3] GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks[C]// IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Las Vegas, USA: IEEE, 2016: 2414-2423.
[4] LI Chuan, WAND M. Combining Markov random fields and convolutional neural networks for image synthesis[C]// IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Las Vegas, USA: IEEE, 2016: 2479-2486.
[5] FIER J, JAMRIKA O, SIMONS D, et al. Example-based synthesis of stylized facial animations[J]. ACM Transactions on Graphics, 2017, 36(4): 155
[6] GATYS L A, ECKER A S, BETHGE M. A neural algorithm of artistic style[J]. Computer Science.(2015- 09- 02)[2018- 01- 01].https://arxiv.org/pdf/1508.06576v2.pdf
[7] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]// Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, USA: Curran Associates Inc, 2012: 1097-1105.
[8] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. Computer Science.(2014- 09- 04)[2018- 01- 01]. https://arxiv.org/abs/1409.1556
[9] GATYS L A, ECKER A S, BETHGE M. Texture synthesis using convolutional neural networks[C]// Proceedings of the 28th International Conference on Neural Information Processing System. [S.l.:s.n.], 2015, 1: 262-270.
[10] SELIM A, ELGHARIB M, DOYLE L. Painting style transfer for head portraits using convolutional neural networks[J]. ACM Transactions on Graphics, 2016, 35(4): 1-18.
[11] SARAGIH J M, LUCEY S, COHN J F. Face alignment through subspace constrained mean-shifts[C]// IEEE 12th International Conference on Computer Vision. Kyoto, Japan: IEEE, 2009: 1034-1041.
[12] BEIER T, NEELY S. Feature-based image metamorphosis[C]// Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques. New York, USA: ACM, 1992: 35- 42.
[13] LIU Ce, YUEN J, TORRALBA A. Sift flow: dense correspondence across scenes and its applications[M]// Dense Image Correspondences for Computer Vision. Berlin: Springer, 2016: 15- 49.
[14] SHIH Y C, PARIS S, BARNES C, et al. Style transfer for headshot portraits[J]. ACM Transactions on Graphics, 2014, 33(4): 1-14.
[15] LUAN F, PARIS S, SHECHTMAN E, et al. Deep photo style transfer[C]// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017: 6997-7005.
[16] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
[17] LEVIN A, LISCHINSKI D, WEISS Y. A closed form solution to natural image matting[C]// Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington D C: IEEE Computer Society, 2006, 1: 61-68.
[18] JADERBERG M, SIMONYAN K, ZISSERMAN A, et al. Spatial transformer networks[J]. Computer Science.(2015- 06- 05)[2018- 01- 01]. https://arxiv.org/pdf/1506.02025v2.pdf

备注

引言

1 具有真实细节的图像风格迁移

2 对空间差异鲁棒的人脸图像风格迁

3 实验结果及分析

4 结语

期刊信息

备注

引言

1 具有真实细节的图像风格迁移

2 对空间差异鲁棒的人脸图像风格迁

3 实验结果及分析

4 结 语

期刊信息

4 结语