In the few years since I wrote my previous blog exploring Visual Arts and AI:The next step towards Computational creativity, the world has seen a surge in interest and experimentation at the intersection of art and artificial intelligence. From using it as a tool for generating art, assisting artists, analyzing art, and even curating exhibitions, it's clear that AI has begun its exploration in our world of colorful splashes!
As someone who has been passionate about both art and AI for quite sometime, I've been fortunate enough to be part of this exciting intersection myself. During my research internship in University, I worked on an industry project that developed an AI tool for blind people that utilized computer vision techniques and spatialized audio and haptics on the outputs of machine learning algorithms.
Around the same time in 2021, I was also working as a software developer at an Art Tech company, where I had the opportunity to build a machine learning pipeline to reproduce paintings. It was fascinating to see the process of teaching an AI model to learn from and imitate the styles of different brushstrokes, and the results were often striking in their accuracy and beauty.
As I marvel at the synergy of these fields, in this new piece of writing, I wish to explore these experiences and insights in more depth, so, let's jump in and see what we can discover together!
Painting is not just a beautiful mode of expression but also that of perception and experience.Every one feels it differently and connects to it in different ways. I have always believed that painting has an edge over other art forms because it transcends geographical boundaries, language barriers, and other limitations other forms may posses but, inaccessible for people with visual impairments.
A lot of tools in the market like screen readers that aids this group of audience are non-interactive, and provide a minimal summary at best, without offering a cognitive understanding of the content. So the shared reality lab at McGill University directed by Prof.Jeremey Cooperstock, decided to provide reach sonification of web graphics to provide users a better experience . I got the opportunity to work on an independent module with this server, using my own paintings as a dataset for my semantic segmentation and object detection models, which is now available as the IMAGE plugin on google chrome. The below example is a beta version of one of my paintings. This project opened my eyes to the immense potential of AI to not only create new art, but also to enhance the experience of experiencing and appreciating art for a wider audience.
Please wear headphones to experience the audio spatialisation - like- for the tree on the right, it is just the right speaker of your headphone that has volume and you can also feel the depth!
The Art tech startup I worked for, was trying to reproduce original paintings with the help of a robot so that we can generate more copies of an artwork as paintings rather than prints so that the consumer market has a more affordable price point and the artist earns better in the process! Apart from using techniques like feature extraction and computer vision techniques, the more important part along with writing scripts to convert strokes to GCode for the hardware was to teach the bot to reproduce a stroke as close as possible to the original one. Simple mapping of co-ordinates with is not a solution as these are not uniform lines from point A to B ! They vary in texture, thickness, opacity and a lot of overlap!
The starting point to solve this actually came from personal experience. As I was pondering on this, I coincidently started a landscape painting titled “Mirror” and it rang a bell ! I learnt different strokes, techniques and styles from different teachers so that made me think on the lines of having to have different models for different kinds and styles of strokes and paintings! It was hard to have one single holistic model and decided to get each stroke right! When it comes to teaching a bot also this holds good.The popular techniques used through this pipeline included neural style transfer, which involves training a neural network to replicate the style of a given brushstroke then we have convolutional neural networks, which can be used to recognize patterns and strokes unique to different styles and then reproduce them accordingly.(I did have a lot of proof of concepts that failed in the process,but thats a part of the game!)My last exploration as a developer there was to have something working instantly, which lead to developing a small generative adversarial networks which is an approach to teach a model to generate similar content based on a given input dataset, and yes the robot did paint something!
From creating new forms of art to enhancing the experience of appreciating art for a wider audience, AI has proven to be a valuable tool in the world of colour and as I mentioned earlier, I have surely been fortunate to work on projects that have explored this intersection. From developing an AI tool for blind people to building a machine learning pipeline to reproduce paintings, these experiences throw light on the immense potential and the infinite possibilities that lie ahead.
Reference and more info about the IMAGE plugin can be found here
If you need some publicly available starter code that deal with above ideas please feel free to reach out to me on Rakshitha@swapnasrushti.org or Rakshitharavi18@hotmail.com and I will be happy to re-explore!
Rakshitha - a very informative post . Thank you for sharing your thoughts, approach and outcomes. True AI is still nascent when it comes to abstract thinking and reproduction. What I liked is the acceptance and exploratory mindset to embrace change to open possibilities.