[1]郭美钦,江健民.人脸图像风格迁移的改进算法[J].深圳大学学报理工版,2019,36(3):230-236.[doi:10.3724/SP.J.1249.2019.03230]
　GUO Meiqin and JIANG Jianmin.Spatially-robust image style transfer for headshot portraits[J].Journal of Shenzhen University Science and Engineering,2019,36(3):230-236.[doi:10.3724/SP.J.1249.2019.03230]
点击复制

人脸图像风格迁移的改进算法()
分享到：

《深圳大学学报理工版》[ISSN:1000-2618/CN:44-1401/N]

卷:
第36卷
期数:
2019年第3期
页码:
230-236
栏目:
【电子与信息科学】
出版日期:
2019-05-20

文章信息/Info

Title:
Spatially-robust image style transfer for headshot portraits
文章编号:
201903002
作者:
郭美钦; 江健民
深圳大学计算机与软件学院，广东深圳 518060
Author(s):
GUO Meiqin and JIANG Jianmin
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060,Guangdong Province, P.R.China
关键词:
计算机图像处理; 图像风格迁移; 卷积神经网络; 归一化互相关; 仿射变换; 语义分割; 人脸图像
Keywords:
computer image processing;  image style transfer;  convolutional neural network;  normalized cross-correlation;  affine transformation;  semantic segmentation;  headshot portrait
分类号:
TP 391
DOI:
10.3724/SP.J.1249.2019.03230
文献标志码:
A
摘要:
图像风格迁移是对给定的输入图像，采用计算机手段自动将任意其他图像的风格迁移到输入图像中，使两者风格完全相同，但保持原图像内容不变．本研究通过实验发现，当这种风格迁移用到人脸图像时，由于嘴唇、眼睛等部位对空间位置差异比较敏感，容易导致风格迁移效果失真．基于DPST（deep photo style transfer）算法，提出对空间差异鲁棒的人脸图像风格迁移的改进算法，通过将输入图像和风格图像各部分语义的分割内容在深度卷积神经网络中特征图通道间的归一化互相关最大化，对风格图像进行对应的仿射变换，每个分割区域都以对应的空间变换后的样式图像作为样式参考，有效减轻了输入图像和风格图像各部分语义分割内容在空间上存在较大差异时产生的重影．选取空间差异较大的输入图像与风格图像，对新算法和原DPST算法以及其他基准算法进行对比实验，结果表明，当两张图像各语义分割内容在空间位置相差较大时，新算法仍能产生较好的效果，鲁棒性更佳．
Abstract:
For an input image, the image style transfer is to change the style of the input image accordingly without modifying its original content. Our empirical studies discover that, when the existing style transfer techniques are applied to headshot portraits, the noticeable distortions could be incurred due to the fact that the relative location of those facial components, such as lips, noses, and eyes, are sensitive to spatial differences. Based on the DPST (deep photo style transfer) method, a spatially-robust image style transfer algorithm for headshot portraits is proposed in this paper to improve the robustness of image style transfer. The proposed algorithm provides an improved solution for the problem that the transferred content could incur ghost shadows when segmented regions between the input image and the style image are significantly different. By maximizing the normalized cross-correlation of the channels of feature maps of corresponding segmented regions of the input image and the style image in a pre-trained convolutional neural network, we propose to apply affine transformations to the style image before it is used for the style transfer. Each segmented region takes the corresponding spatially transformed style image as style reference, making it adaptive to the spatial difference between the input and the style image and the incurred ghost shadow can be minimized or eliminated. The experimental results show that our proposed algorithm can obtain good results even when the semantic segmentation between two images have the large spatial differences. The new algorithm also displays the better robustness than the DPST algorithm and other benchmark.

参考文献/References:

［1］ REINHARD E, ASHIKHMIN M, GOOCH B, et al. Color transfer between images［J］. IEEE Computer Graphics and Applications, 2002, 21(5): 34-41.
［2］ SUNKAVALLI K, JOHNSON M K, MATUSIK W, et al. Multi-scale image harmonization［J］. ACM Transactions on Graphics, 2010, 29(4): 125.
［3］ GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks［C］// IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE, 2016: 2414-2423.
［4］ LI Chuan, WAND M. Combining Markov random fields and convolutional neural networks for image synthesis［C］// IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE, 2016: 2479-2486.
［5］ FIER J, JAMRIKA O, SIMONS D, et al. Example-based synthesis of stylized facial animations［J］. ACM Transactions on Graphics, 2017, 36(4): 155
［6］ GATYS L A, ECKER A S, BETHGE M. A neural algorithm of artistic style［J］. Computer Science. (2015-09-02)［2018-01-01］.https://arxiv.org/pdf/1508.06576v2.pdf
［7］ KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks［C］// Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, USA: Curran Associates Inc, 2012: 1097-1105.
［8］ SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［J］. Computer Science. (2014-09-04)［2018-01-01］. https://arxiv.org/abs/1409.1556
［9］ GATYS L A, ECKER A S, BETHGE M. Texture synthesis using convolutional neural networks［C］// Proceedings of the 28th International Conference on Neural Information Processing System. ［S.l.:s.n.］, 2015, 1: 262-270.
［10］ SELIM A, ELGHARIB M, DOYLE L. Painting style transfer for head portraits using convolutional neural networks［J］. ACM Transactions on Graphics, 2016, 35(4): 1-18.
［11］ SARAGIH J M, LUCEY S, COHN J F. Face alignment through subspace constrained mean-shifts［C］// IEEE 12th International Conference on Computer Vision. Kyoto, Japan: IEEE, 2009: 1034-1041.
［12］ BEIER T, NEELY S. Feature-based image metamorphosis［C］// Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques. New York, USA: ACM, 1992: 35-42.
［13］ LIU Ce, YUEN J, TORRALBA A. Sift flow: dense correspondence across scenes and its applications［M］// Dense Image Correspondences for Computer Vision. Berlin: Springer, 2016: 15-49.
［14］ SHIH Y C, PARIS S, BARNES C, et al. Style transfer for headshot portraits［J］. ACM Transactions on Graphics, 2014, 33(4): 1-14.
［15］ LUAN F, PARIS S, SHECHTMAN E, et al. Deep photo style transfer［C］// IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017: 6997-7005.
［16］ CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
［17］ LEVIN A, LISCHINSKI D, WEISS Y. A closed form solution to natural image matting［C］// Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington D C: IEEE Computer Society, 2006, 1: 61-68.
［18］ JADERBERG M, SIMONYAN K, ZISSERMAN A, et al. Spatial transformer networks［J］. Computer Science. (2015-06-05)［2018-01-01］. https://arxiv.org/pdf/1506.02025v2.pdf
［19］ KINGMA D P, BA J. Adam: a method for stochastic optimization［J］. Computer Science. (2014-11-22)［2018-01-01］. https://arxiv.org/pdf/1412.6980v8.pdf

相似文献/References:

[1]殷夫,纪震,周家锐,等.PET/CT图像质量主观评价与感知模型[J].深圳大学学报理工版,2015,32(2):205.[doi:10.3724/SP.J.1249.2015.02205]
　Yin Fu,Ji Zhen,Zhou Jiarui,et al.Subjective assessment and perception model of PET/CT image quality[J].Journal of Shenzhen University Science and Engineering,2015,32(3):205.[doi:10.3724/SP.J.1249.2015.02205]
[2]贾志成,郑笑,郭艳菊,等.改进鲸群优化子空间匹配追踪的稀疏解混算法[J].深圳大学学报理工版,2020,37(1):63.[doi:10.3724/SP.J.1249.2020.01063]
　JIA Zhicheng,ZHENG Xiao,GUO Yanju,et al.Sparse unmixing using the improved whale optimized subspace matching pursuit algorithm[J].Journal of Shenzhen University Science and Engineering,2020,37(3):63.[doi:10.3724/SP.J.1249.2020.01063]

备注/Memo

备注/Memo:
Received：2018-05-10；Revised：2018-08-28；Accepted：2018-09-11
Foundation：Key Program for International Cooperation of National Natural Science Foundation of China (61620106008)；Shenzhen Basic Research Funding Project (JCYJ20160226191842793)
Corresponding author：Professor JIANG Jianmin．E-mail: jianmin.jiang@szu.edu.cn
Citation：GUO Meiqin，JIANG jianmin．Spatially-robust image style transfer for headshot portraits［J］. Journal of Shenzhen University Science and Engineering, 2019, 36(3): 230-236.（in Chinese）
基金项目：国家自然科学基金委国际合作重点资助项目(61620106008)；深圳市学科布局资助项目(JCYJ20160226191842793)
作者简介：郭美钦(1992—)，深圳大学硕士研究生．研究方向：图像处理、深度学习．E-mail：guomeiqin2017@email.szu.edu.cn
引文：郭美钦，江健民．人脸图像风格迁移的改进算法［J］. 深圳大学学报理工版，2019，36(3)：230-236.
更新日期/Last Update: 2019-04-22