Prompting Ain’t Easy
A look at AI as it enters commercial creative adolescence
Over the course of 2023, AI tools went mainstream. It will be known as the year that AI exploded onto the scene. As a global society we are wrestling with some of the largest questions about how artificial intelligence will impact our existence. Is this the dawn of a new era of human productivity when we begin to solve the most complex problems that have always been out of our intellectual reach? Or are we entering into the first stages of our eventual downfall as a species where the machines take over? These are really intense things to ponder as we tinker with and explore new AI-powered tools while we also continue to do all we can to do great work.
I agree with John Battelle that in 2024 we will see AI get its sea legs and further mature while every government and industry works through issues such as privacy, ownership, data sources, reskilling, and adoption. Visual designers in branding, graphic design, and marketing are required to keep pace with technological evolution so that we can do the work at hand. Over the course of our creative working lives it is inevitable that we adopt, master, and abandon dozens of software solutions and processes as a natural extension of being creative and curious and remaining employable and relevant. Designers are always naturally learning and reskilling. We don’t need a corporate mandate or a class assigned by leadership to learn and evolve. We just do it.
For most creatives, adopting AI tools has been no different from using any other new creative software or process. You download, research, play around, and make stuff. Over time, technologies become normalized and you forget when they were ever not in your life. Smartphones, cloud storage, smart TVs, streaming music services, and online shopping did not exist 20 years ago yet all are now normal facets of daily life. The same will happen with AI, but it will be more embedded, dispersed, and in most cases invisible. Eventually it will feel the way having electric power feels now. Ubiquitous, taken for granted, and only noticeable when it’s not available.
Getting AI tools to do what you want them to do
In mid-January 2023 I went on a trip with my dad to Antarctica and Patagonia — a big item on his bucket list. As we crossed the deep swells of the Drake Passage, I was fussing around on Midjourney over spotty satellite Wi-Fi trying to make a scene that merged 1970 Le Mans Porsche 911s and a Mars landscape. It took a lot of prompting and refinement but I got some stuff I liked. I realized I had to really stick with it, be organized with my prompting structure, and analyze how subtle changes in the prompt affected the output. If I worked at it I could move the needle from just guessing to something closer to planning the end result.
Fast forward a mere 12 months and we have seen vast improvement in the level of sophistication of what can be achieved and a global proliferation of tools. As I started writing this at the end of 2023, we entered a new world where everyone is starting to use AI tools in our day-to-day work. How do we move from playing around in a novel and sometimes haphazard way to settling into behaviors that are native to what we do every day? AI tools are still evolving at the same time we are working to learn and use them. What I could get out of ChatGPT-3 is very different from what I can get out of ChatGPT-4 now. There are also competing platforms like Bard, Otter, Bing AI, Claude, and many more evolving in real time.
AI tools that change every time you use them make adoption and mastery very difficult. In contrast, the adoption of a hardware product like the iPhone back in 2007–2008 was seamless. It was a phone, a Web browser, and an iPod in one device. Adoption was easier because we understood its usefulness based on its similarity to previous devices. With AI, there is no precedent for training an AI on our proprietary data or for learning the correct prompting syntax to produce an image in Midjourney, DALL-E, or Stable Diffusion, for example. These are entirely new tasks that we have to make a concerted effort to learn, adopt, and normalize. We can’t just swipe our finger on a screen. It takes learning and focused work.
Because the knowledge and behaviors required to make using AI tools effective in your day-to-day job are by no means intuitive or frictionless, it is often a very messy process of trial and error to get results that you feel are worth the effort. For most designers and creatives, messy things that can produce happy accidents are the kinds of tools we love. For others who require more predictability in their work, getting AI tools to do what you want them to do can seem almost impossible. It is this huge divide between the efficiency and advantages that AI promises and the individual and collective investment of knowledge and changes in behavior required to get there that will remain tumultuous for the foreseeable future.
AI for frictionless and intuitive use
Tools that are powered by AI behind the scenes that “just work” are already all around us and will continue to proliferate. A good example is letsenhance.io, which does a good job of enhancing images. When you’re pressed for time, remove.bg is very effective at removing the backgrounds of images. Chatbots can help with anything from a question about banking to information about car insurance. We use AI for countless things easily and intuitively because the people who developed them made them intuitive to use. Millions of people have been yelling at Siri and Alexa for information, recipes, and song lyrics for years. We will continue to talk to AI-powered intelligence to help us in every area of our lives. Sooner than we think, the qualifier “powered by AI” will no longer be relevant because it will not be a differentiator. In the near future, we’ll find it strange if we can’t communicate with our ATM machine, the product scanner at the grocery store, our insurance website, or any point of service where AI assistance would help us and save us time.
Learning and training AI for your work and making
We will certainly continue to have AI tools that “just work” but we will also need teams to learn, train, and use AI tools in ways that will require new skills and new ways of thinking. It is in these areas that a great deal of change will continue to occur as certain tasks get replaced by AI tools entirely, other skills are augmented by AI to become easier, and others remain harder to master and integrate. Workplace productivity tools like Notion (of which I’m a huge fan) have already integrated AI natively. The next phase of workplace productivity is using proprietary and public APIs to train AI to assist organizations in ways that are bespoke to their needs and datasets.
Over the last year I’ve been using Midjourney and have also played around with DALL-E and Stable Diffusion. A few of my colleagues have been exploring these tools and we have found them all to be powerful and fun to use. There is also an inherently imprecise nature to them that you have to accept and embrace. Developing the right text prompt to get the image you want is an exercise in research, repetition, and refinement. If you are looking for more control or a specific style, gesture, head position, or anything nuanced you may be hard pressed to get it. When you are tinkering it is fun to see what kind of image you can produce. I see the value in using tools like Midjourney to make images and AI art to share with friends. When it comes to client work these tools can be really useful if you know how to wield them. I’ve used text to image AI to create assets for mood boards and have had success creating abstract 3D compositions to use in brand explorations. AI tools are great for creative augmentation because you can generate more ideas and then pick and choose what is useful to you, be it verbal or visual.
The most successful use of AI-generated imagery, motion, and language in our day-to-day design and brand development practice are the ones that feel invisible and native to the larger context in which they are integrated and appear. We can still usually tell when something is AI generated — especially visuals, because each technology has a certain look. In Midjourney you have to use the –style parameter to suppress the app’s default style and Style Tuner to dial in a look you want. It takes a second to get these kinds of parameters to do what you want them to do. Stable Diffusion using LoRA models allows you to make more focused changes in character, style, concept, pose, clothing, and object. The more you can fine tune, the better the result. For a recent project we utilized AI-generated imagery to avoid spending time finding existing imagery or having to purchase it. For those willing to take the time and iterate, it’s a game changer. You can make your own photography, icons, and 3D assets from prompts. Similarly, you can refine a process for training copy tone. The goal is to get the magical AI thing to look like a normal thing and to work seamlessly with the rest of your brand system.
From infinite possibilities to rigorous systems
The applied AI research company Runway is an interesting harbinger of the future. Its software has been focusing on generating short videos and changing existing videos from text prompts and images. In 2023, Runway raised $141 million in funding from Alphabet Inc., Nvidia Corp., Salesforce, and other investors. Runway’s AI Magic Tools allow users to leverage generative AI for content creation in the form of images and video. Using generative AI to augment your creative process is becoming standard practice. What is not standard are the methods and ways of working. Right now a motivated creative has dozens if not hundreds of possible AI tools to play with. “Play” is the operative term here because there are no rigorous standards or methods. We are living in a massive generative AI remix playground that is as fun as it is chaotic.
Because of the wide variety of tools available, we are seeing a kind of creative hopscotch between them to get different results. For example, I recently was using Midjourney to generate an asset. It was not getting me what I wanted so I jumped to ChatGPT-4 with DALL-E integrated. I uploaded the Midjourney image, which looked kind of like what I wanted, and then worked in ChatGPT-4 to dial it in with further prompting. I could have done the same process in something like Image Creator from Microsoft Designer to yield other visualizations. Once I had an image that I liked, I could then pop it into Adobe Illustrator, outline it, and prompt Illustrator to create different vector-based visualizations based on that style. This sounds a bit crazy and disjointed, but the novelty of context switching among AI tools and seeing what will happen makes it fun and engaging. It is a kind of hyper powered trial and error.
The road ahead is will be more personalized and organized
What is on the horizon is more standardization for how generative AI tools are leveraged and which ones large organizations deem legitimate, safe, and free of copyright issues for creative teams to learn and use. OpenAI has been leading the AI race. Launched in November 2023, Open AI’s GPT Store allows people to make their own tailored versions of ChatGPT. GPTs are amazingly powerful allowing you to create task-specific AI assistants without coding. One look at the GPT Store and you can see everything from a GPT to help you book your vacation, another that can make a Pixar-style avatar of you from a photo, and everything in between. Simple novelty GPTs live alongside much more complex ones. In the realm of graphic design and branding, ChatGTP already has a Canva plugin that allows users to enter text prompts to create generic designs. The Canva plugin takes your text prompt and pulls in Canva’s existing templates. It is not very nuanced — it can’t create custom typography or custom graphics like DALL-E or Midjourney. But it’s a start.
My presumption is that Adobe Creative Suite and Figma will come out with their own AI tools that will allow designers to train them to connect systems thinking with design iteration. We are getting close to having the ability to use AI solutions like ChatGPT and others to create our own chatbots that we can train to help visualize designs that adhere to clear formal rules. We are probably six months to a year out from being able to train AI to create high fidelity variations on brand systems after they have been trained on the brand’s parameters. The tricky part in all of this will still be training AI to adhere to the visual and verbal parameters of a brand system and to have them also understand the nuances of the use cases needed. Any brand’s visual and verbal system is unique to it. The form factors it must take, from marketing communications to digital products, can be incredibly varied, from button to billboard. In the short term, having a designer effectively train an AI to behave correctly and adhere to a system may actually be more difficult, and less cost effective, than having talented creative people make the thing itself. When we use tools like Midjourney and ChatGPT we never submit a text prompt and get exactly what is in our heads. The question I ask myself is in what situations does training AI to behave within design parameters make smart business sense and when is it a potential exercise in futility, eating up time and causing frustration. Promptin’ ain’t easy. Training chatbots to follow rules is still a skillset that requires a good bit of motivation, patience, and learning.
In 2024 nothing is fully baked.
There is a deep joy and satisfaction that comes with using AI to generate visual ideas that would normally take hours if not days. Using AI for creative augmentation is here to stay, and I’m all for it. When brand building, it is better to have more raw material to edit down than to have too little to evaluate against. AI brings yet another incredibly powerful tool into our creative toolbox, but it’s not a replacement for our individual creative authorship. I’m very excited by all the possibilities AI offers, but it can be a bit exhausting. In the near term we are still working to demystify and normalize AI in creative practice. A few big players will certainly emerge as the new standard. For now we must be patient while we also continue to learn, reskill, iterate, and make a lot more happy accidents. Wherever we are a year from now, I know it will be mind-blowing.
Thanks for reading. If you are interested, check out my book A Visible Distance: Craft, Creativity and the Business of Design.
Subscribe here to be notified when I publish.
