Amateurs figure out how to insert custom fonts into AI-generated images

Last week, an amateur experimenting with the new Flux AI image synthesis model found that it is unexpectedly good at rendering custom-trained reproductions of typefaces. While far more efficient ways to display computer fonts have existed for decades, the new technique is useful for AI image hobbyists because Flux is able to render accurate depictions of text, and users can now feed words rendered in custom fonts directly into the AI ​​image generation.

We've had the technology to create precisely smooth computer-rendered fonts in custom shapes since the 1980s (1970s in the research field), so creating an AI-replicated font is not big news in itself. But a new technique means you can see a particular font in images generated by AI, such as a chalkboard menu in a photorealistic restaurant or a printed business card held by a cyborg fox.

Shortly after the emergence of mainstream AI image synthesis models like Stable Diffusion in 2022, some people started wondering: How can I insert my own product, clothing item, character, or style into an image generated by AI? One answer that emerged came in the form of LoRA (low-rank adaptation), a technique discovered in 2021 that allows users to augment the knowledge in an AI base model with custom-trained modular add-ons.

These LoRAs, as the modules are called, allow the image synthesis model to create new concepts that are not originally found (or are poorly represented) in the base model's training data. In practice, image synthesis hobbyists use these to represent unique styles (e.g., everything in chalk art) or themes (e.g., detailed images of Spider-Man). Each LoRA must be trained specifically using user-provided examples.

Before Flux, most AI image generators weren't very good at rendering accurate text in a scene. If you prompted Stable Diffusion 1.5 to render a sign that said “thing,” it would return gibberish. OpenAI's DALL-E 3, released last year, was the first mainstream model to do text fairly well. Flux still makes occasional mistakes with words and letters, but it's the most capable AI model at rendering “in-world text” (whatever you might call it) that we've seen so far.

Since Flux is an open model available for download and fine-tuning, it might be a wise move to train the typeface Lora first used last month. This is exactly what was done to train a typeface Lora. AI enthusiast Vadim Fedenko (who did not respond to a request for an interview by press time) recently discovered this. “I'm really impressed with how this turned out,” Fedenko wrote in a Reddit post. “Flux recognizes what characters look like in a certain style/font, making it possible to train Lorus with specific fonts, typefaces, etc. Looking forward to training more soon.”

For his first experiment, Fedenko chose a bubbly “Y2K” style font, reminiscent of fonts popular in the late 1990s and early 2000s, and published the resulting model on the Civitai platform on August 20. Two days later, a Civitai user named “AggravatingScree7189” posted a second typeface, LoRA, which is similar to the font. cyberpunk 2077 Video games.

Responding to Fedenko's post on the Y2K font, a Reddit user named Eggs-Benadrill wrote, “The text was so bad I never thought you could do that.” Another Redditor wrote, “I didn't know the Y2K journal was fake until I zoomed in on it.”

Is this an exaggeration?

in great shape , An example of this cyberpunk 2077 LoRA, presented with Flux Dev.

It's true that using a deeply trained image synthesis neural network to render a plain old font on a simple background is probably overkill. You probably wouldn't want to use this method to replace Adobe Illustrator when designing a document.

“It looks cool, but it's weird how we're reinventing the idea of ​​fonts as 300MB LoRAs,” one commenter on Reddit wrote on a thread about this. cyberpunk 2077 Font.

Generative AI is often criticized for its environmental impact, and this is a legitimate concern for massive cloud data centers. But we find that Flux can insert these fonts into AI-generated scenes while running in quantized (size-reduced) form locally on an RTX 3060 (and the full dev model can run on an RTX 3090). This is the same power consumption as playing a video game on the same PC. The same applies to LoRA creation: cyberpunk 2077 Font trained LoRA in three hours on a 3090 GPU.

There are also ethical issues with using AI image generators, such as how they are trained on data collected without the consent of the content owner. Even though this technology is divisive among some artists, a large community of people use it every day and share the results online through social media platforms like Reddit, leading to new applications of this type of technology.

At the time of this writing, there are only two custom Flux typefaces LoRA, but we've already heard of people planning to create more. Although it's still in its early stages, the technology for creating typefaces LoRA could become the foundation when AI image synthesis is more widely implemented in the future. Adobe, with its own image synthesis model, is likely keeping an eye on this.