Being natural to everyone, language-based inputs have demonstrated effective for various tasks such as object detection and image generation. This paper for the first time presents a language-based system for interactive colorization of scene sketches, based on their semantic comprehension. Compared with prior scribble-based interfaces, which require a minimum level of professional skills, our language-based interface is more natural for novice users. The proposed system is built upon deep neural networks trained on a large-scale repository of scene sketches and cartoon-style color images with text descriptions. Given a scene sketch, our system allows users, via language-based instructions, to interactively localize and colorize specific object instances to meet various colorization requirements in a progressive way. We demonstrate the effectiveness of our approach via comprehensive experimental results including alternative studies, comparison with the state-of-the-art, and generalization user studies.
VIDEO
Publications ACM Transactions on Graphics (SIGGRAPH Asia), 2019.
Keywords: deep neural networks; image segmentation; language-based editing; scene sketch; sketch colorization, interactive graphics, interactive perception, augmented communication
European Conference on Computer Vision (ECCV), 2018.
Keywords: sketch dataset, scene sketch, sketch segmentation, interactive graphics
Cited By Sketch-Based Creativity Support Tools Using Deep Learning Artificial Intelligence for Human Computer Interaction: A Modern Approach. Forrest Huang , Eldon Schoop , David Ha , Jeffrey Nichols , and John Canny . source | cite | search Emergent Graphical Conventions in a Visual Communication Game https://arxiv.org/pdf/2111.14210.pdf. Shuwen Qiu , Sirui Xie , Lifeng Fan , Tao Gao , Song-Chun Zhu , and Yixin Zhu . source | cite | search Write-an-Animation: High-level Text-based Animation Editing with Character-Scene Interaction Computer Graphics Forum. Jia-Qi Zhang , Xiang Xu , Zhi-Meng Shen , Ze-Huan Huang , Yang Zhao , Yan-Pei Cao , Pengfei Wan , and Miao Wang . source | cite | search One Sketch for All: One-Shot Personalized Sketch Segmentation https://arxiv.org/pdf/2112.10838.pdf. Anran Qi , Yulia Gryaditskaya , Tao Xiang , and Yi-Zhe Song . source | cite | search FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context https://arxiv.org/pdf/2203.02113.pdf. Pinaki Nath Chowdhury , Aneeshan Sain , Yulia Gryaditskaya , Ayan Kumar Bhunia , Tao Xiang , and Yi-Zhe Song . source | cite | search Delaunay Painting: Perceptual Image Colouring From Raster Contours with Gaps Computer Graphics Forum. Amal Dev Parakkat , Pooran Memari , and Marie-Paule Cani . source | cite | search SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs IEEE Transactions on Image Processing. Fang Liu , Xiaoming Deng , Changqing Zou , Yu-Kun Lai , Keqi Chen , Ran Zuo , Cuixia Ma , Yong-Jin Liu , and Hongan Wang . source | cite | search A Review of Image and Video Colorization: From Analogies to Deep Learning Visual Informatics. Shu-Yu Chen , Jia-Qi Zhang , You-You Zhao , Paul L. Rosin , Yu-Kun Lai , and Lin Gao . source | cite | search Exemplar-Based Sketch Colorization with Cross-Domain Dense Semantic Correspondence Mathematics. Jinrong Cui , Haowei Zhong , Hailong Liu , and Yulu Fu . source | cite | search Neural Image Recolorization for Creative Domains 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Boyi Li , Serge Belongie , Ser-nam Lim , and Abe Davis . source | cite | search SketchMaker: Sketch Extraction and Reuse for Interactive Scene Sketch Composition ACM Transactions on Interactive Intelligent Systems. Fang Liu , Xiaoming Deng , Jiancheng Song , Yu-Kun Lai , Yong-Jin Liu , Hao Wang , Cuixia Ma , Shengfeng Qin , and Hongan Wang . source | cite | search Modulating Bottom-Up and Top-Down Visual Processing Via Language-Conditional Filters 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Ilker Kesen , Ozan Arkan Can , Erkut Erdem , Aykut Erdem , and Deniz Yuret . source | cite | search Unsupervised Scene Sketch to Photo Synthesis arXiv.2209.02834. Jiayun Wang , Sangryul Jeon , Stella X. Yu , Xi Zhang , Himanshu Arora , and Yu Lou . source | cite | search Cartoon Image Processing: A Survey International Journal of Computer Vision. Yang Zhao , Diya Ren , Yuan Chen , Wei Jia , Ronggang Wang , and Xiaoping Liu . source | cite | search DrawMon: A Distributed System for Detection of Atypical Sketch Content in Concurrent Pictionary Games Proceedings of the 30th ACM International Conference on Multimedia. Nikhil Bansal , Kartik Gupta , Kiruthika Kannan , Sivani Pentapati , and Ravi Kiran Sarvadevabhatla . source | cite | search Generating Compositional Color Representations From Text https://arxiv.org/abs/2109.10477. Paridhi Maheshwari , Nihal Jain , Praneetha Vaddamanu , Dhananjay Raut , Shraiysh Vaishay , and Vishwa Vinay . source | cite | search Painting Style-Aware Manga Colorization Based on Generative Adversarial Networks 2021 IEEE International Conference on Image Processing (ICIP). Yugo Shimizu , Ryosuke Furuta , Delong Ouyang , Yukinobu Taniguchi , Ryota Hinami , and Shonosuke Ishiwatari . source | cite | search Adversarial Segmentation Loss for Sketch Colorization 2021 IEEE International Conference on Image Processing (ICIP). Samet Hicsonmez , Nermin Samet , Emre Akbas , and Pinar Duygulu . source | cite | search Focusing on Persons 4. Xin Jin , Zhonglan Li , Ke Liu , Dongqing Zou , Xiaodong Li , Xingfan Zhu , Ziyin Zhou , Qilong Sun , and Qingyu Liu . source | cite | search Sketchy Scene Captioning: Learning Multi-level Semantic Information From Sparse Visual Scene Cues Lecture Notes in Computer Science. Lian Zhou , Yangdong Chen , and Yuejie Zhang . source | cite | search Multi-style Chinese Art Painting Generation of Flowers IET Image Processing. Feifei Fu , Jiancheng Lv , Chenwei Tang , and Mao Li . source | cite | search TediGAN: Text-Guided Diverse Face Image Generation and Manipulation Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Weihao Xia , Yujiu Yang , Jing-Hao Xue , and Baoyuan Wu . source | cite | search DanbooRegion: An Illustration Region Dataset Computer Vision –ECCV 2020. Lvmin Zhang , Yi Ji , and Chunping Liu . source | cite | search SketchyDepth: From Scene Sketches to RGB-D Images ICCV 2021. Gianluca Berardi , Samuele Salti , and Luigi Di Stefano . source | cite | search Generative Adversarial Networks\textendashEnabled Human\textendashArtificial Intelligence Collaborative Applications for Creative and Design Industries: A Systematic Review of Current Approaches and Trends Frontiers in Artificial Intelligence. Rowan T. Hughes , Liming Zhu , and Tomasz Bednarz . source | cite | search XCI-Sketch: Extraction of Color Information From Images for Generation of Colored Outlines and Sketche https://arxiv.org/abs/2108.11554. Harsh Rathod , Manisimha Varma , Parna Chowdhury , Sameer Saxena , V. Manushree , Ankita Ghosh , and Sahil Khose . source | cite | search Generating Compositional Color Representations From Text https://arxiv.org/abs/2109.10477. Paridhi Maheshwari , Nihal Jain , Praneetha Vaddamanu , Dhananjay Raut , Shraiysh Vaishay , and Vishwa Vinay . source | cite | search Text As Neural Operator:Image Manipulation by Text Instruction MM '21: Proceedings of the 29th ACM International Conference on Multimedia. Tianhao Zhang , Hung-Yu Tseng , Lu Jiang , Weilong Yang , Honglak Lee , and Irfan Essa . source | cite | search DLA-Net for FG-SBIR 5. Jiaqing Xu , Haifeng Sun , Qi Qi , Jingyu Wang , Ce Ge , Lejian Zhang , and Jianxin Liao . source | cite | search Grayscale Image Colorization Using a Convolutional Neural Network Journal of the Korean Society for Industrial and Applied Mathematics. Minje Jwa and Myungjoo Kang . source | cite | search Exploring Local Detail Perception for Scene Sketch Semantic Segmentation IEEE Transactions on Image Processing. Ce Ge , Haifeng Sun , Yi-Zhe Song , Zhanyu Ma , and Jianxin Liao . source | cite | search Partially Does It: Towards Scene-Level FG-SBIR with Partial Input arXiv.2203.14804. Pinaki Nath Chowdhury , Ayan Kumar Bhunia , Viswanatha Reddy Gajjala , Aneeshan Sain , Tao Xiang , and Yi-Zhe Song . source | cite | search ColorizeDiffusion: Adjustable Sketch Colorization with Reference Image and Text arXiv.2401.01456. Dingkun Yan , Liang Yuan , Yuma Nishioka , Issei Fujishiro , and Suguru Saito . source | cite | search OpenSketch: A Richly-annotated Dataset of Product Design Sketches ACM Transactions on Graphics. Yulia Gryaditskaya , Mark Sypesteyn , Jan Willem Hoftijzer , Sylvia Pont , Frédo Dur , and Adrien Bousseau . source | cite | search Language-based Photo Color Adjustment for Graphic Designs ACM Transactions on Graphics. Zhenwei Wang , Nanxuan Zhao , Gerhard Hancke , and Rynson W.H. Lau . source | cite | search Semi-supervised Reference-based Sketch Extraction Using a Contrastive Learning Framework ACM Transactions on Graphics. Chang Wook Seo , Amirsaman Ashtari , and Junyong Noh . website , source | cite | search Stroke-based Semantic Segmentation for Scene-level Free-hand Sketches The Visual Computer. Zhengming Zhang , Xiaoming Deng , Jinyao Li , Yukun Lai , Cuixia Ma , Yongjin Liu , and Hongan Wang . source | cite | search Deep Learning for Studying Drawing Behavior: A Review Frontiers in Psychology. Benjamin Beltzung , Marie Pel{\'{e}} , Julien P. Renoult , and C{\'{e}}dric Sueur . source | cite | search Human\textendashmachine Hybrid Intelligence for the Generation of Car Frontal Forms Advanced Engineering Informatics. Yu Wu , Lisha Ma , Xiaofang Yuan , and Qingnan Li . source | cite | search SVCNet: Scribble-based Video Colorization Network with Temporal Aggregation arXiv.2303.11591. Yuzhi Zhao , Lai-Man Po , Kangcheng Liu , Xuehui Wang , Wing-Yin Yu , Pengfei Xian , Yujia Zhang , and Mengyang Liu . source | cite | search Adversarial Interactive Cartoon Sketch Colourization with Texture Constraint and Auxiliary Auto-Encoder Computer Graphics Forum. Xiaoyu Liu , Shaoqiang Zhu , Yao Zeng , and Junsong Zhang . source | cite | search Fine-Grained Video Retrieval with Scene Sketches IEEE Transactions on Image Processing. Ran Zuo , Xiaoming Deng , Keqi Chen , Zhengming Zhang , Yu-Kun Lai , Fang Liu , Cuixia Ma , Hao Wang , Yong-Jin Liu , and Hongan Wang . source | cite | search A State-of-Art Review on~Intelligent Systems for~Drawing Assisting Lecture Notes in Computer Science. Juexiao Qin , Xiaohua Sun , and Weijian Xu . source | cite | search Improving Sketch Colorization Using Adversarial Segmentation Consistency arXiv.2301.08590. Samet Hicsonmez , Nermin Samet , Emre Akbas , and Pinar Duygulu . source | cite | search CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing arXiv.2402.17624. Chufeng Xiao and Hongbo Fu . source | cite | search Learning Inclusion Matching for Animation Paint Bucket Colorization arXiv.2403.18342. Yuekun Dai , Shangchen Zhou , Qinyue Li , Chongyi Li , and Chen Change Loy . source | cite | search CreativeSeg: Semantic Segmentation of Creative Sketches IEEE Transactions on Image Processing. Yixiao Zheng , Kaiyue Pang , Ayan Das , Dongliang Chang , Yi-Zhe Song , and Zhanyu Ma . source | cite | search 
