LLM OS / OpenAI / Andrej Karpathy 2023 Nov Week 5 Datumo Newsletter |
|
|
π° Meta's video generation model 'Emu' |
|
|
Language generation model 'LLaMA' and image model 'Emu'" image. DALL-E 3.
In mid-November, while the 'OpenAI Incident' involving the firing of CEO Sam Altman unfolded, Meta introduced new features of its in-house image generation foundation model 'Emu (Emu)'.
These include 'Emu Video', which generates videos based on text, and 'Emu Edit', which edits images. Emu is used in Meta's Instagram image editing tools and the Imagin feature, which generates images from messages. In this letter, we will take a closer look at 'Emu Video'.
*Emu (Enhancing Image Generation Models Using Photogenic Needles in a Haystack) is Meta's first image generation foundation model, unveiled at this year's 'Meta Connect'. It is a diffusion model pre-trained on 1.1 billion image-text pairs, then fine-tuned with thousands of high-quality images.
The name Emu refers to a type of bird resembling an ostrich. It seems similar yet quite different from the text foundation model 'LLaMA' :)
|
|
|
The 'Emu Video' generates videos in two stages. |
|
|
A panda bear driving a car.' Meta AI, Emu Video
"The recently unveiled 'Emu Video' is a text-to-video (t2v) generation model that creates videos based on input text. While the dominant paradigm in traditional video creation used diffusion models to generate all video frames at once, Meta researchers have divided the creation process into two steps.
First, generate an image that fits the input prompt, and 2) then generate a video referencing both the created image and the input prompt. This method borrows from the way language models in natural language processing (NLP) generate sentences. *Casual Language Models, including GPT, continue to generate the next word based on the previously predicted words.
|
|
|
First, generate an image 'I' according to the text 'p',
and then use the generated image and text to create a video 'V'.
"Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning"
In Emu Video, this method is applied to video generation. Instead of sequentially generating each frame of the video, it first creates a complete image and then generates the rest of the video.
This approach allows AI models to make more accurate predictions by generating video frames sequentially based on an already completed image. The Meta researchers stated in their paper, 'Intuitively, providing the model with a starting image and text makes video generation easier since the model only needs to predict how the image will evolve in the future.'
|
|
|
Human evaluators compared the EMU VIDEO with traditional video generation models
According to the researchers, this method of generation surpasses direct T2V methods even when considering the same amount of training data, computing resources, and trainable parameters.
In actual performance tests, Emu Video, which uses only two diffusion models to create a 4-second video, significantly outperformed existing video generation models in human comparison evaluations.
Especially, Emu Video surpassed Meta's last year's 'Make-A-Video' by 96% in 'video quality' and 85% in 'fidelity to text prompts', and also performed better than commercial solutions like Gen2 and Pika Labs.
Experience the demo at the link :)
|
|
|
The Data for Smarter AI
Datumo is an all-in-one data platform for your smarter AI.
|
|
|
Datumo Inc.
π contact@datumo.com
|
|
|
|
|
