Transforming The World of AI With Sora: Text-to-Video Generator

Last year, OpenAI revolutionised the world of artificial intelligence with ChatGPT. This year, OpenAI has introduced the world with Sora which is an AI video generator. Open AI has trained the model using publicly available and copyrighted licensed-for-purpose videos. The sources of video to train Sora have not been disclosed by OpenAI.  Let us learn about this generative AI model in detail. 

You must have used generative AI tools such as Midjourney, Leonardo.AI, Synthesia among several others, Sora AI is the game changer. This generative model will change a lot of things including filmmaking. From creating an AI video showing Will Smith eating noodles, we have come a long way ahead. While there are already some tools like Runway ML that can animate still images; Sora has been able to create videos that are highly realistic though they are not perfect as of now. In the coming years, these models will only become more realistic.

Table of Contents

• What is Sora?
• How to use Sora?
• Who Can Use Sora?
• Features of Sora
• How Videos Are Turned into Patches?
• Capabilities of Sora: What Can It Do?
• Limitations of Sora
• How Does Open AI’s Sora Work?

What is Sora AI?

Sora AI is a text-to-video generator introduced by OpenAI. 'Sora' is a Japanese word that means 'sky' and it signifies limitless creative potential. The model can generate high-fidelity videos of up to 1 minute based on text instructions. The technology behind Sora is the adaption of the technology used in DALL-E 3. In Sora, users input a prompt and the model generates videos based on it. 

Sora-generated videos are tagged with C2PA metadata to show that these are created using artificial intelligence model. OpenAI has provided sample videos by Sora on its website. These videos are not modified and are exactly what the model has created.

Explore artificial intelligence courses

Here is an animated version of GTA 6 created with Sora:

How to Use Sora?

At present, the exact process of using Sora is not determined since it is available to only a few users for now. These users are testing the tool for its capabilities and usage.

Who Can Use Sora?

As of February 2024, the model is available for use to:

• Red teamers so that they can detect vulnerabilities, harms and risks. 
  A small team of designers, visual artists and filmmakers to gain feedback on how this model can be upgraded for creative professionals.

Check out VFX courses

Features of Sora

Open AI’s text-to-video generator Sora has the following features:

• Generates high-quality videos that accurately follow user prompts.
• Can sample vertical 1080 x 1920p and widescreen 1920 x 1080p videos and any resolution in between.
• Can generate videos with dynamic camera motion. As the camera shifts, scene elements and people consistently move through the 3D plane.
• Generates images as well in variable size upto 2048 x 2048 resolution.
• Creates content for different devices directly at the native aspect ratio.
• Quickly allows prototype content at lower sizes before generating full-resolution video.
• Videos created with Sora have improved composition and framing. 
• Can perform image and video editing tasks. This includes animating static images, creating perfectly looping videos and extending videos backwards and forwards in time.
• It can interpolate between two input videos which allows seamless transitions between videos having different subjects and scene compositions. Here is a screenshot showing interpolation (the middle image shows interpolation of left and right image).

• Sora can keep videos consistent over time. It understands how things in a video are connected in the short term and long periods. Sora can remember and track people, animals, and objects in a video, even if they get blocked from view or move off-screen. Here is a video generated based on the prompt 'The story of a robot’s life in a cyberpunk setting'. As the camera pans left and right, it shows the same placement of objects. 

• It can effectively model short as well as long-range dependencies. 
• Sora can generate multiple shorts of the same character in the video maintaining the same appearance of the character throughout the video. Here is one sample video based on the prompt: "A movie trailer featuring the adventures of the 30-year-old space man wearing a red wool knitted motorcycle helmet, blue sky, salt desert, cinematic style, shot on 35mm film, vivid colors." Notice how the same character appears throughout the video.

• It can simulate artificial processes such as video games such as Minecraft.

Capabilities of Sora: What Can It Do?

Open AI's Sora is capable of performing the following based on the text prompt:

• Generate videos from the text prompt provided by users. Below is a sample video based on the prompt.
• Generates an entire video from existing still images or even fills in missing frames. Here is an example (at 0:46) 

• Generates complex scenes with multiple characters.

• Automatically creates different video angles without being prompted.
• Accurately generates the details of the subject of the video and background.

• Creates multiple shorts within a single video

Limitations of Sora

While Sora is quite advanced compared to other text-to-video generators, it still has several limitations:

• It cannot accurately follow the laws of physics in complex scenes such as eating food or glass shattering. 
• Certain objects may appear out of nowhere. Take for instance the below-shown video based on the prompt "Five gray wolf pups frolicking and chasing each other around a remote gravel road, surrounded by grass. The pups run and leap, chasing each other, and nipping at each other, playing.". As you can see, the prompt mentions 5 pups. However, during the video, more pups appear from nowhere.
  

• It may create physically implausible motions. One such video has been generated through the prompt "Step-printing scene of a person running, cinematic film shot in 35mm."

• Sora can confuse spatial details of a prompt and struggle with a precise description of an event that takes place over time.

• Simulation of complex interactions between multiple characters and objects is difficult for Sora at present. As you can see in the below video, there are several issues. The hand movements of some characters are unnatural. One of the candle's wicks has two flames. Also, the facial movement of the main character seems flawed. 

How Does Open AI’s Sora Work?

Here is how Open AI's Sora works:

1. Data Collection: Sora is trained on a large dataset of videos and images of variable durations, resolutions, and aspect ratios. This dataset provides the model with diverse examples of visual content to learn from.
2. Latent Code Extraction: The videos and images in the dataset are processed to extract latent codes, which represent the underlying features and patterns in the visual data. These latent codes capture both spatial and temporal information.
3. Transformer Architecture: Sora utilizes a transformer architecture, a type of neural network that is effective at capturing long-range dependencies in data. The transformer operates on spacetime patches of video and image latent codes, allowing it to understand complex temporal dynamics and spatial relationships.
4. Text-Conditional Training: Sora is trained in a text-conditional manner, meaning that it generates videos based on textual input. The model learns to generate video sequences that align with the given text prompts, enabling control over the content and style of the generated videos.
5. Unsupervised Learning: Sora learns representations of video sequences through unsupervised learning techniques. By training on a diverse set of video and image data without explicit labels, the model can capture the underlying structure and patterns in the visual content.
6. Video Generation: Once trained, Sora is capable of generating new, high-fidelity videos based on text prompts. The model takes the text input, processes it through the transformer architecture, and generates a minute of realistic video content that aligns with the provided text description.
7. Evaluation and Fine-Tuning: The generated videos are evaluated for fidelity, coherence, and realism. Fine-tuning may be performed to improve the quality of the generated videos further.

You can visit openai.com/sora to know more about the model.