SKETCHES are created by humans through an iterative process and reflect
one’s sketching skills, taste, world perception and even character IN JUST A SET OF SPARSE LINES.
Being the result of semantic, perceptual, or conceptual processing, sketches are distinctive from photos.
While the Computer Vision and Machine Learning communities have firmly invested in reasoning with
photos, sketch data just recently got into the spotlight. This shift of focus on using sketch data has
already started to cause a profound impact on many facets of research on computer vision, computer graphics,
machine learning, HCI and artificial intelligence at large. Sketch has not only been used for image
RETRIEVAL, 3D MODELING, USER INTERFACE DESIGN, but also as a key enabler
in our fundamental understanding of VISUAL ABSTRACTION,
CREATIVITY, and EXPRESSIVITY.
Developing such understanding is impossible with photos.
This (series of) workshop aims to BRING TOGETHER RESEARCHERS FROM INTERDISCIPLINARY RESEARCH FIELDS
in the community to consolidate cross-discipline insights, IDENTIFY AND ENCOURAGE NEW RESEARCH DIRECTIONS,
and ultimately foster the growth of the sketch research community.
Dr. Metin Sezgin graduated summa cum laude with Honors from Syracuse University in 1999. He completed his MS in the Artificial Intelligence Laboratory at Massachusetts Institute of Technology in 2001. He received his PhD in 2006 from Massachusetts Institute of Technology. He subsequently joined the Department of Computer Science at the University of Cambridge as a Postdoctoral Research Associate. Dr. Sezgin is currently an Associate Professor in the College of Engineering at Koç University, Istanbul. His research interests include intelligent human-computer interfaces, multimodal sensor fusion, and HCI applications of machine learning. Dr. Sezgin is particularly interested in applications of these technologies in building speech- and pen-based intelligent interfaces. Dr. Sezgin held visiting posts at Harvard University and Yale University. He is an associate editor of the IEEE Transactions on Affective Computing, and the Computers & Graphics journal. His research has been supported by international and national grants including grants from the European Research Council, and Turk Telekom. He is a recipient of the Career Award of the Scientific and Technological Research Council of Turkey. As a consultant, Dr. Sezgin led technical teams in a diverse range of industries, including automotive, banking, defense, telecom, and retail.
Dr. Adrien Bousseau is a researcher at Inria Sophia-Antipolis in the GraphDeco research group. He did his Ph.D. at Inria Rhône-Alpes and his postdoc at UC Berkeley. He also did several internships at Adobe Research. Adrien does research on image creation and manipulation, with a focus on drawings and photographs. Most notably he worked on image stylization, image editing and relighting, vector graphics, and sketch-based modeling. He received one of the three Eurographics 2011 Ph.D. award for his research on expressive image manipulations, and a young researcher award from the French National Research Agency (ANR) for his work on computer-assisted drawing. He received an ERC Starting Grant to work on drawing interpretation for 3D design.l Research Agency (ANR) for his work on computer-assisted drawing. Adrien received an ERC Starting Grant in 2016 to work on drawing interpretation for 3D design.
Dr. Masha Shugrina is a Senior Research Scientist at the NVIDIA Toronto AI Lab, where she manages a group focused on creative applications of AI and efforts to accelerate research. Her key research interest is enabling AI to work in the interactive loop, enhancing rather than replacing creativity. She has pursued her passion for creativity-enhancing applications and robust software engineering in many other settings. She defended her PhD at the University of Toronto, where she investigated the design playful and intelligent creative tools, receiving Canada's Alain Fournier Award for Outstanding Doctoral Dissertation in Computer Graphics and founding colorsandbox.com. Prior to this, she was a Research Engineer at Adobe Research (Cambridge, MA), where she led the Playful Palette project. Before Adobe, Masha got her Master’s from MIT, where she worked on customizable models for 3D printing. Before returning to graphics research, Masha had an established engineering career as a Senior Software Engineer / Tech Lead at Google (NYC and Zurich). She is also an avid oil painter, whenever she can find the time.
Dr. Edgar Simo-Serra is currently an associate professor at Waseda University. He obtained his Industrial Engineering degree from BarcelonaTech in 2011 and his Ph.D. in 2015 from the same university. From 2015 to 2018 he was at Waseda University as a junior researcher (assistant professor), and during 2018 he was a JST Presto Researcher before rejoining Waseda.
His general research interests are in the intersection of computer vision, computer graphics, and machine learning with applications to large-scale real world problems.
Prof. Cheng Deng received the B.Sc., M.Sc., and Ph.D. degrees in signal and information processing from Xidian University, Xi’an, China. He is currently a Full Professor with the School of Electronic Engineering at Xidian University.
His research interests include multimodal machine learning, computer vision, and deep learning. He is the author and coauthor of more than 130 scientific articles at top venues, including IEEE T-PAMI, T-NNLS, T-CYB, T-MM, T-SMC, T-IP, ICML, NeurIPS, ICCV, CVPR, IJCAI, and AAAI. He has served as a reviewer or a program committee member to more than 10 leading computer science conferences including ICML, NeurIPS, CVPR, ICCV, KDD, and more than 30 leading international journals including IJCV, IEEE T-PAMI, T-CYB, T-NNLS, T-IP, T-CSVT, T-MM, T-GRS, T-IFS, T-KDD, Information Sciences, Information Fusion, Pattern Recognition, Signal Processing, etc. He is currently serving on the editorial boards of Pattern Recognition, Neurocomputing, and Pattern Recognition Letters, and served ICCV2021 and CVPR2021 as an Area Chair.
Prof. Yi-Zhe Song is a Professor of Computer Vision and Machine Learning, and Director of SketchX Lab at the Centre for Vision Speech and Signal Processing (CVSSP), University of Surrey. He obtained a PhD in 2008 on Computer Vision and Machine Learning from the University of Bath, a MSc (with Best Dissertation Award) in 2004 from the University of Cambridge, and a Bachelor's degree (First Class Honours) in 2003 from the University of Bath. He is an Associate Editor of the IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), and Frontiers in Computer Science – Computer Vision. He served as a Program Chair for the British Machine Vision Conference (BMVC) 2021, and regularly serves as Area Chair (AC) for flagship computer vision and machine learning conferences, most recently at ECCV’22 and CVPR'22. He is a Senior Member of IEEE, a Fellow of the Higher Education Academy, as well as full member of the EPSRC review college.
Keynote session: 30 minutes talk + 10 minutes Q&A.
Paper session: 10 minutes talk + 5 minutes Q&A.
| Time (Tel-Aviv: GMT+3) | Session | Speaker | Chair |
| 2:00 PM - 2:05 PM | Welcome / Opening Remarks | Qian Yu | |
| 2:05 PM - 2:45 PM | Keynote talk: "Sketching, Illustration, and Machine Learning" Abstract:
Since the dawn of civilization, humans have always been drawing to convey emotions, represent abstract concepts, commemorate events, or transmit knowledge. From cave paintings, to modern digital painting software, technology has always assisted in the illustration process. In this talk, I shall introduce and focus on recent machine learning techniques for aiding line drawing and digital illustration. I will cover a myriad of existing research to deal with the particularities of line drawings, and also present open problems and outstanding issues in the field.
|
Edgar Simo-Serra | Richard Zhang |
| 2:45 PM - 3:25 PM |
Keynote talk: "From sketch, to CAD, to sketch" Abstract:
Computer Aided Design (CAD) is a multi-billion dollar industry responsible for the digital creation of almost all manufactured goods. Central to CAD productivity is the concept of parametric 3D modeling, which allows relevant dimensions of a shape to be changed after its conception. However, the gain of productivity promised by parametric models is strongly diminished by the extreme difficulty of creating such models in the first place. To promote effective modeling strategies, design educators and practitioners advocate freehand sketching as a preliminary step to parametric modeling. Our key insight is to consider freehand sketching as the natural language of designers, which needs to be translated into the formal language of parametric computer-aided design. In this talk, I will present a series of projects that aims at automating the translation of freehand sketches into parametric CAD models, and vice-versa.
|
Adrien Bousseau | Mikhail Bessmeltsev |
| 3:25 PM - 4:05 PM | Keynote talk: "Cross-domain Sketch-based Modeling: Retrieval and Visual Generation" Abstract:
Hand-drawn sketches can express users' ideas intuitively and flexibly, and are easy to interact with mobile devices, which have attracted extensive attention in the fields of image retrieval and visual content generation. Currently, most methods employ generative adversarial networks to design models across two domains of sketch and natural image for these downstream tasks. This talk first summarizes the existing popular methods and their shortcomings, and introduces the research progress in sketch-based retrieval and visual content generation from two cross-domain perspectives, i.e., data and semantic, and finally discusses the future research in this field.
|
Cheng Deng | Yonggang Qi |
| 4:05 PM - 4:45 PM |
Keynote talk:
"A Critical Review of Sketch Collection Methods: Remembering How Humans Really Sketch" Abstract:
Sketching is a natural and effortless mode of expression that doesn't require specialized knowledge. As a powerful tool for processing and communicating ideas, sketching promises immense opportunities in the field of Human-Computer Interaction (HCI). That's why the community has long been devoted to building sketch recognition systems to introduce this modality into our relationship with technology. However, with the increased need for data to train intelligent systems, sketch collection practices have inclined towards standardized methods that take no account of natural sketching behaviors. Today, researchers are blinded by an abundance of sketch data that misrepresent real sketches, and consequently build models that are ineffective in real-life applications. We would like to raise awareness of the community's deviation from ideal utilization of sketches by emphasizing the discrepancy between natural sketches and those presented in the literature. In our work, we enumerate features associated with sketching observed in daily life, present a critical review of popular sketch datasets, and promote good practices to encourage community in building realistic sketch collection settings that will yield natural sketch data.
|
Metin Sezgin | Giorgos Tolias |
| 4:45 PM - 5:00 PM |
Invited paper: Sketch2Pose: estimating a 3D character pose from a bitmap sketch Brodt, K. and Bessmeltsev, M., ACM TOG'22 |
Kirill Brodt | Xiaoguang Han |
| 5:00 PM - 5:15 PM |
Invited paper: Learning to generate line drawings that convey geometry and semantics Chan, C., Durand, F. and Isola, P, CVPR'22 |
Caroline Chan | Qian Yu |
| 5:15 PM - 5:55 PM |
Keynote talk: "Vision != Photo" Abstract:
While the vision community is accustomed to reasoning with photos, one
does need to be reminded that photos are mere raw pixels with no
semantics. Recent research has recognised this very fact and started
to delve into human sketches instead -- a form of visual data that had
been inherently subjected to human semantic interpretation. This shift
has already started to cause profound impact on many facets of
research on computer vision, computer graphics, machine learning, and
artificial intelligence at large. Sketch has not only been used as
novel means for applications such as cross-modal image retrieval, 3D
modelling, forensics, but also as key enablers for the fundamental
understanding of visual abstraction and creativity which were
otherwise infeasible with photos. This talk will summarise some of
these trends, mainly using examples from research performed at
SketchX. We will start with conventional sketch topics such as
recognition, synthesis, to the more recent exciting developments on
abstraction modelling and human creativity. We will then talk about
how sketch research has redefined some of the more conventional vision
topics such as (i) fine-grained visual analysis, (ii) 3D vision
(AR/VR), and (iii) OCR. We will finish by highlighting a few open
research challenges to drive future sketch research.
|
Yi-Zhe Song | Stella Yu |
| 5:55 PM - 6:35 PM |
Keynote talk: "Designing Creative Tools In the Age of AI" Abstract:
Last year has seen staggering progress in AI-driven generation of visual content. But how do we leverage AI to design tools that truly support creativity, beyond loosely controlled exploration? In this talk, I will go over core considerations when designing for the creative process, including control, input modalities, exploration and serendipity. I will include several case studies from research projects from the Creative and Applied AI Tools (CAAT) group at NVIDIA Toronto AI Lab, where we focus on researching, developing and launching AI technology as real tools creators can use. Research covered will include generative 3D models and generative modeling of interactive drawing tools. More than anything this talk is intended to give food for thought for designing creative tools in the modern technology landscape.
|
Maria Shugrina | Yulia Gryaditskaya |
| 6:35 PM - 6:40 PM | Closing remarks | Yulia Gryaditskaya |
Dr. Yulia Gryaditskaya is an Assistant Professor (Lecturer) in Artificial Intelligence at Surrey Institute for People-Centred AI and CVSSP, UK. Prior to that, she was a Senior Research Fellow (2020-2022) in Computer Vision and Machine Learning at the Centre for Vision Speech and Signal Processing (CVSSP), in the SketchX group, led by Prof. Yi-Zhe Song. Before joining CVSSP, she was a postdoctoral researcher (2017-2020) at Inria, GraphDeco, under the guidance of Dr. Adrien Bousseau. She had the opportunity to visit MIT and collaborate with Prof. Fredo Durand, MIT, CSAIL, and Prof. Alla Sheffer, UBC, British Columbia. She received her Ph.D. (2012-2016) from MPI Informatik, Germany, advised by Prof. Karol Myszkowski and Prof. Hans-Peter Seidel. While working on her Ph.D. (in 2014), she did a research internship in the Color and HDR group in Technicolor R&D, Rennes, France, under the guidance of Dr. Erik Reinhard. She received a degree (2007-2012) in Applied Mathematics and Computer Science with a specialization in Operation Research and System Analysis from Lomonosov Moscow State University, Russia.
Yulia's research is on how AI can be used to facilitate creative process and help boost human creativity and expressivity in the context of sketching and 3D modeling.
Dr. Qian Yu is an associate professor at Beihang University. Before joining Beihang, she was a postdoctoral research fellow at UC Berkeley / ICSI, 2018-2019, working with Dr. Stella Yu. She received her Ph.D. degree from the Queen Mary University of London in 2018, advised by Dr. Yi-Zhe Song and Prof. Tao Xiang. Her research is on computer vision and deep learning, focusing on human sketch understanding, including synthesis, recognition, and related applications.
Qian has published her work on sketch understanding at top computer vision journals and conferences, such as IJCV, CVPR, ICCV, and ECCV. Her work on sketch recognition, ‘Sketch-a-Net that Beats Humans’, was awarded as the ‘Best Scientific Paper’ at BMVC 2015.
Dr. Yonggang Qi is an assistant professor (lecturer) at Beijing University of Posts and Telecommunications (BUPT), Beijing, China. He received his PhD degree in Signal Processing at BUPT in 2015. From 2019 to 2020, he was a visiting scholar at SketchX Lab at the Centre for Vision Speech and Signal Processing (CVSSP) in University of Surrey. He also worked as a guest PhD at Aalborg University in Denmark in 2013 and a visiting researcher at Sun Yat-sen University in China in 2014. His research interests include perceptual grouping, and sketch-based vision tasks such as sketch-based image retrieval (SBIR), sketch recognition, sketch generation and language-based sketch understanding.
His research interests include perceptual grouping and sketchbased vision tasks, and he has published over 10 sketch papers, including at CVPR, ICCV, TIP, TCSVT etc.
Dr. Stella Yu is the Director of Vision Group at the International Computer Science Institute, a Senior Fellow at the Berkeley Institute for Data Science, and on the faculty of Computer Science, Vision Science, Cognitive and Brain Sciences at UC Berkeley.
Dr. Yu is interested not only in understanding visual perception from multiple perspectives, but also in using computer vision and machine learning to automate and exceed human expertise in practical applications.
Dr. Giorgos Tolias is an Assistant Professor at CTU in Prague and is leading a team within the Visual Recognition Group. Previously, he was a post-doc at Inria, Rennes. His work got the Best Science Paper Award - Honorable Mention at BMVC 2017 and has served as an AC for ECCV 2020. He co-organized workshops on Visual Instance-Level Recognition at prior major computer vision conferences such as CVPR, ECCV, and ICCV.
Giorgos’s research interests include sketch recognition on which he has published at CVPR’17, ECCV’18, and IVC’18.
Dr. Mikhail Bessmeltsev is an Assistant Professor at Universite de Montreal, Quebec Canada. He did his postdoc at MIT (CSAIL) with Justin Solomon in Cambridge, MA. Before, he completed his Ph.D. in Computer Science at the University of British Columbia (Vancouver, Canada). Before that (2004-2010) he did his Bachelor’s and Master’s at Mechanics & Mathematics Department of Novosibirsk State University (Akademgorodok, Novosibirsk, Russia).
Sketching lies at the center of his interests, with a focus on sketch-based modeling and beautification. His work on sketching is regularly published at ACM TOG, CGF, and ICLR.
Dr. Xiaoguang Han is an Assistant Professor, a presidential young fellow at the Chinese University of Hong Kong, Shenzhen, and a research scientist at Shenzhen Institute of Big Data. He obtained a Ph.D. degree in computer science from The University of Hong Kong in 2017. His papers were selected as best paper finalists of CVPR 2019 and paper award nominees of CVPR 2020.
His work on deep sketch-based 3D modeling and sketch-based image synthesis is published at top graphics conference and journals, such as ACM TOG and Pacific Graphics, and top UI conference UIST.
Dr. Richard Zhang is a Senior Research Scientist at Adobe Research, with interests in computer vision, deep learning, machine learning, and graphics. He obtained his PhD in EECS, advised by Professor Alexei A. Efros, at UC Berkeley in 2018. He graduated summa cum laude with BS and MEng degrees from Cornell University in ECE. He is a recipient of the 2017 Adobe Research Fellowship.