■ Synaesthetic Strokes
Liat Grayver
Collegium Helveticum, ETHZ Zürich 2023
It is my physical knowledge, the embodied artistic experience I have gained from over 20 years of professional involvement in various forms of painting and traditional printmaking, that has led me to focus on the exploration of the brushstrokes as a key element in the development of our global and local cultural and perceptual æsthetic history. The brushstroke, in all its diverse forms, serves as a singular tool of communication encountered in paintings and drawings across different epochs. At the heart of this artistic practice is a profound exploration of the act of creating a painting itself, emphasizing the process of crafting a line rather than the final artwork.
Synaesthetic Strokes utilizes artificial learning systems that are inspired by the structure and functionality of biological neural networks to explore how the body and mind perceive not only the visual elements of the painting (the image), but also the intricate processes by which it was brought into existence — capturing both the form perceived by the eye and the energy felt within. Throughout the process of artistic exploration, passive materials respond to the active manipulation imposed upon them. Both the passive and active elements play equally significant roles in the creative process and the final outcome. Each aspect along the way must be meticulously assessed in order to faithfully recreate a brushstroke. Factors such as brush deformation, ink state, overlapping strokes, velocity, tilt, vectors and pressure all contribute to the physical properties that are etched into the visual manifestation of the stroke itself. Inspired by the principles of Japanese calligraphy, this work seeks to strike a delicate balance between controlling the materials and embracing bodily movements, adapting to the idiosyncrasies of the paper and ink as they inform the calculation of the subsequent stroke.
The system pipeline begins with a human creating a brushstroke using a calligraphic brush, ink and a large roll of mulberry paper measuring 1 x 60 m. The machine then attempts to reproduce the stroke by analyzing data captured from a Dynamic Vision Sensor (DVS) camera, initially focusing on the stroke’s direction and velocity. As the process unfolds, the machine incorporates additional data from the force sensor and robotic arm to generate its own stroke. To enable a continuous painting experience and showcase the evolutionary state of the stroke, I have implemented a specially designed paper holder and magnetic frame. This dynamic setup allows growth of the painting and provides the human operator with the ability to subjectively determine when the paper is sufficiently saturated or æsthetically pleasing. At this point I pause the painting process, scroll down to a clean section of the paper and allow the machine to continue generating new strokes. This interactive setup empowers the artist to actively engage with the machine in shaping the artistic output, highlighting the collaborative nature of the artwork. It combines the
precision and capabilities of the machine with the subjective decisions and artistic sensibilities of the human operator, resulting in a unique and evolving artistic creation. Upon painting approximately 5–7 metres of paper, painting was cut from the long scroll and suspended from the ceiling to the floor of the historical building at the ETH observatory, where the exhibition was held. This arrangement gradually filled the entire room with a combination of mechanical and human-made strokes, immersing visitors in a sea of brushstrokes and challenging them to discern between those created by the machine and those by a human. The overlapping of strokes, whether dry on wet or wet on wet, results in the formation of intricate and complex compositions, 1 m wide and up to 7 metres in length. This composition emerges from a fusion of algorithmic procedures, the intricate behaviours inherent in the materials and the machine’s perceptive understanding of the previous stroke. The outcome is an organic evolution of strokes, embodying the dynamic interplay between human creativity and decision-making in conjunction with machine processes.
The resulting performative artwork embodies a dynamic and collaborative exploration between humans and artificial systems, deconstructing the essence of a brushstroke and provoking enquiries into its nature. It prompts reflection on the perception and replication of brushstrokes by artificial entities, as well as the profound significance of the technical complexities inherent in this seemingly simple task. These challenges bear relevance not only in the
context of rapid advancements in autonomous machines but also in understanding the capabilities and boundaries of human agency.
The primary artistic objective of this project was to develop an adaptive artificial behaviour system that incorporates principles of traditional craftsmanship to create robotic-based paintings. These paintings serve as a foundation for training the machine to achieve complex and nuanced outcomes and reactions throughout the painterly process. The long-term goal of this project is the creation of a brushstroke depository, or library, of “gestural” robotic movements (tactile-physical) and presets of rules and “conditions” for their execution (perceptual-cultural), an accessible platform that will be built later for broader use in robotic painting (both hardware and software). As the
library will compartmentalize not only individual brushstrokes as gestures, but also visual information, it will be possible to integrate the repository into the visual feedback mechanism of another robotic painting interface. As an example of the many important implications this holds for the future of robotic painting, this will afford users with the possibility to decide on the level of human control or autonomous robotic painting creation, as it will be possible
to set rules for the use of individually defined variants of stroke characteristics. This exploration aimed to redefine the traditional art form of painting in the context of our technology-driven era, while examining the influence of robots as interactive painterly tools on creativity, authorship and agency in artificial systems.
Lorem ipsum dolor sit amet, consetetur sadipscing
During a 10-month Junior Fellowship (September 2022 – June 2023) at the Collegium Helvetica at the ETH Zurich,I developed the artwork Synaesthetic Strokes, in which I sought to shed new light on the ever-growing relationship between human expressivity and mechanical operations. The technical development was done by Lioba Schürmann under the supervision of Prof. Dr. Giacomo Indiveri from the Institute of Neuroinformatics at the University of Zurich (UZH) and the ETH Zurich. Additionally, valuable guidance was given by Dr. Giulia D’Angelo and Prof. Dr. Chiara Bartolozzi from the Italian Institute of Technology. This exploration aimed to investigate the dynamic interplay between humanity and its media, forging a deeper understanding of the intricate connection between human expressiveness, creativity and technological innovation — from the traditional practices of manual craftsmanship to
autonomous machineoperation.
Building upon Ursula Franklin’s definition of holistic and prescriptive technologies8, within the artistic practice of creating and conceptualizing the installation and performative artwork Synaesthetic Strokes I was particularly interested in working adaptively to an ever-changing world of æsthetic exploration, with the tasks and outcome coexisting more as an open-end relational process than a set of fixed rules. In contrast, thus far, my robotic painting
work was dependent on custom-developed computer programs, which have constrained some of the æsthetic avenues I wished to explore, as the limitations were defined by specific code architectures. The utilization of neuromorphic sensory processing technology, a biology-inspired artificial computing system, presented an intriguing opportunity to delve into the realm of interactive robotics. This technology offered a temporally based system that deviated from fixed algorithmic procedures and limited datasets, providing a more adaptive approach for exploring the art of brushstrokes in the context of robotics-assisted painting.
In the creation of the work Synaesthetic Strokes, I developed novel methodologies for creating an active perception system that facilitates creative generative processes. This is achieved through the integration of a dynamic vision sensor for visual feedback and a neuromorphic chip for sensory processing and pattern recognition, seamlessly connected to a robotic arm with a force sensor, operating in real time. The focus of this exploration lies in the attempt to comprehend and replicate the essence of a single brushstroke.
In my pursuit of an intrinsically collaborative and interactive working process between humans and machines, I aim to achieve a profound level of synergy. In this vision, the machine not only responds to the visual state of the painting but also delves into the tactile realm of materials, extracting information from the brush and paper themselves. It then adapts its physical implementation of the following stroke accordingly.
Revisiting the task of recreating a brushstroke using artificial systems that incorporate DVS cameras, force sensors and neuromorphic processors has granted direct access to real-time data. This significant development makes it possible to create and work with a new range of dynamic operations, which are at the core of the new work, Synaesthetic Strokes. This newfound capability has not only enhanced the machine’s “understanding” of the intricate aspects required to recreate a stroke but has also allowed me to assess the stroke’s unfolding state, fostering a deeper level of physical collaboration akin to classical craftsmanship, where the affordances, impacts and potentialities of the human, medium and machine components comprising the system are intimately intertwined.
The creation of Synaesthetic Strokes was only possible following the establishment of a new and innovative robotic painting studio during my fellowship at the Collegium Helveticum, combining meticulous planning and coordination of the engineering and artistic concepts. Support from several institutions at ETH Zurich and UZH played a crucial role in the process. The technical setup involved a UR10 Robot, on loan from the Institut für Technologie in der Architektur and Gramazio Kohler Research Group at the ETH Zurich. The robot was equipped with a SensONE 6-axis force torque sensor, provided by Bota Systems, boasting a serial interface for seamless integration. A tripod was utilized to position a web camera and a Dynamic Vision Sensor (DVS) camera14 positioned in front of the painting area. The Neuromorphic DYNAP-SE1 Processor, created by Prof. Giacomo Indiveri at the Institut für Neuroinformatic UZH, was crucial in analyzing the brushstrokes using data captured by the DVS. The processor consists of mixedsignal silicon neuron circuits designed for emulating spiking neural networks, and was used to generate motor commands that made it possible to recreate similar but not identical brushstrokes, matching their timing and dynamics. The robotic painting setup included a custom-built paper holder, produced by the workshop of the UZH Physics Institute, that was connected to an aluminum frame with a magnet-based fixation mechanism. This allowed for continuous painting, affording the human operator control to reposition the roll so that the robot has a blank (clean) section to paint on as needed. Additionally, the setup involved a lighting studio system from the Zürcher Hochschule der Künste (ZHdK) that ensured a stable environment for visual data collection. A 3D-printed brush holder, designed by Zurich-based Design Engineer Carl P-Conquilla, held two different sizes of calligraphic brushes, which make it possible for the system to produce an extraordinarily wide range of brushstroke characteristics. The setup also featured a Plexiglas palette with six glass containers, holding sumi ink or blue ink, white ink, and two containers of water for each of them for dilution or to create colour gradients. To explore different surface resistances and transparency effects, Chinese Xuan rice paper and Mulberry paper were employed.
