Creating Incredible AI-Powered Art With Dall-E 2

Delve into the realm of DALL-E 2, OpenAI's groundbreaking AI art generator. Discover how it transforms text descriptions into vivid, high-resolution images with its CLIP and GLIDE models. Unleash your creativity, explore the training process, and learn how to wield its power to create or modify images effortlessly.

If you are wondering what Dall-E 2 is, how it works, and the ways to use it, we will cover all about this amazing image creator in the most layman’s terms. 

What is Dall-E 2?

OpenAI’s generative text-to-image AI system Dall-E 2 creates realistic, original, and high-resolution images from descriptions in natural language. It can even modify any image you provide and create variations of it.

Founded in 2022 as an extension of its previous AI system Dall-E, it has been named after French Surrealist, Salvador Dalí and the robot character from the famous film – WALL-E. Dall-E couldn’t provide higher-resolution images and did not have much in-depth comprehension as Dall-E 2. A user can also now use the in-painting feature by fixing or altering their own images within the new version.

Dall-E 2 is trained on 650 million images with two main deep learning models – CLIP (Contrastive Language-Image Pre-training) and the Diffusion-based model, GLIDE (Guided Diffusion).

The CLIP model learns what textual semantics represent visually by contrasting images with labels or captions instead of predicting them. This model can also capture style besides semantics. 

The GLIDE model extends the concept of diffusion by adding textual information. A typical Diffusion model generates data by destroying trained data and successively adding Gaussian noise. GLIDE augments this with textual information. 

If you are taking artificial intelligence courses you will have a better understanding of CLIP and GLIDE as discussed in simple terms below.

How Does DALL-E 2 Work?

At its core, it generates images based on suitable captions you provide. It can even edit your images. 

So what goes on behind the scenes? Let’s find out. 

CLIP

In one OpenAI’s research paper, ‘Hierarchical Text-Conditional Image Generation with CLIP Latents‘, the study proposes a method of generating images from basic text descriptions using a model called CLIP. 

CLIP is a neural network model that understands the relationship between text and images. It matches images to the captions instead of simply classifying images available across the internet. 

But it generates images only in stages. Each stage of image generation requires a text description. And the internet is filled with them. Think of Instagram or Pinterest, where there are millions of images with captions. 

Let’s take an example of a text description that states a ‘red flower with five petals’. The CLIP model can generate a rough outline of the flower. It will then add detail based on the text. 

CLIP generates latent vectors that represent text descriptions. The vectors become the input to a generator network that produces the images. 

CLIP is highly advanced in terms of how it can classify images based on random texts. It ensures that the image and text are close to each other.

GLIDE (Guided Diffusion)

GLIDE is an image generation model. It goes a step more than a pure diffusion model. It uses CLIP text embeddings to support the image creation process. 

But let’s take a step back and know what a diffusion model is exactly. 

It is a generative model that takes a piece of information (in this case, an original image) and gradually adds noise to it over time steps to the point it becomes unrecognisable. After that point, it reconstructs the image like it was in the original. This is how it learns how to generate realistic images.

So, where does GLIDE come into the picture?

“GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models” is a research paper that describes a new method for generating and editing images using natural language descriptions.

The authors propose a method called GLIDE (Guided Diffusion). It combines two powerful techniques. One is text-based image synthesis and the other, diffusion models. This text-based synthesis allows users to describe an image using natural language, while the diffusion models learn the underlying patterns in the data and generate photorealistic images.

The GLIDE method can then be used to generate images of fictional characters or scenes based on text descriptions. It can also be used to edit existing images by changing their attributes, such as colour or shape, based on natural language instructions.

Let’s consider an example. 

Suppose you want to generate an image of a red apple on a white background. You can input the following natural language description into GLIDE: “An image of a red apple on a white background.” GLIDE will then use the diffusion models to generate an image that matches this description.

Similarly, you can use GLIDE to edit an existing image. If you have an image of a blue car and want to change its colour to red. You can input the following natural language description into GLIDE: “Change the colour of the car to red.” It will then use the diffusion models to modify the image and produce a new version of the car with a red colour.

An Overview of the Training Process in Dall-E 2

The basics of CLIP and GLIDE are hopefully now clear. There are two more aspects to Dall-E 2. Have a look at the image below.

Let’s look at the relationship between prior and decoder steps. The text you enter transforms into a CLIP text embedding, while below the dotted line, you have the process of creating an image from text that goes through a diffusion prior. Following that, it becomes an image embedding. This continues into the decoder step, where the image embedding becomes the final image. The final image also goes through an upscaling process from 64×64 to 1024x 1024, using Convolutional Neural Network.

How to Use Dall-E 2?

Head over to the OpenAI website and log in with your credentials to create an account. You will see that you have 15 free credits each month and if you have used up, you will have to pay. But don’t worry, for the first month there are 50 free credits. 

Once there, you can autogenerate an image using the Surprise Me option, where the prompt will be random and so will be the AI-generated art. 

Or you can enter a prompt as you prefer. 

Try making the text prompt as descriptive as possible with this AI art generator. 

In the example image below, the description was ‘an astronaut on mars sitting and chilling on an armchair in a photorealistic way’. 

If you didn’t prefer this image, try its variations. Once you get the variations, you can open them on new tabs. Right-clicking on them will give you more options. Refer to the screenshot below. 

You can edit the image, which is currently in beta testing mode. Or, you can select any of the four variations apart from the original to generate more to your liking. 

Apart from that, you can upload an image and try editing it. You can modify it and skim through four variations before accepting it. In the below example, the prompt is the same as on the first image above, but it is slightly modified. It goes “an astronaut in mars on an arm chair with a strange creature drinking wine”.

As you can see, it extends the image. But, don’t forget to use the adjustable features below. You can erase a part from the image like you rasterise on Adobe Photoshop, speaking of which, do check the graphic design courses based on Adobe family.

Dall-E 2 stands out from its competitors Midjourney and Stable Diffusion on the photo editing aspect. You can add your own images and combine it with Dall-E 2’s images – it’s endless!

But before you go, check out the blogs on Dall-E 2’s main competitor, Midjourney, by clicking on the links below.