State-of-the-Art Generative AI 📰 How Microsoft Sees the Future of Gen AI |
|
|
Since the emergence of Stable Diffusion in July 2022, generative models have become a global topic of discussion. In particular, Diffusion-based image generation models have gained significant attention in the vision field. Fast forward two years, and generative models are now capable of creating not just images, but videos and 3D objects as well.
In today’s newsletter, we will explore how generative foundation models are evolving in three key aspects, based on a presentation by a principal researcher at Microsoft at the CVPR 2024 Tutorial. 🧐 |
|
|
High-quality data is essential for training AI models. However, the process of data collection and labeling is no easy task, and has historically come with significant costs. Of course, even when money is invested, it's still far from a straightforward task.
|
|
|
The three images above highlight how descriptions can vary, as people tend to explain things differently. Even when a large number of people are involved, creating consistent data can be challenging due to differences in standards and explanation styles.
One task that AI has taken over is annotation. For Text-to-Image generation models, descriptions in text need to be paired with related images. In order for AI to perform annotation, it must "understand" images in terms of text. To reduce costs while gathering large amounts of text, researchers have used a method of collecting web images along with their associated alternative text (Alt Text). However, there's a problem. As seen in the examples above, Alt Text sometimes contains URLs or random numbers, which can negatively affect the model, making it better to remove such data.
Efforts to build better data have continued, especially since generative models must produce results that meet the diverse needs of users for practical applications. For instance, with the first image, while earlier descriptions might have been brief, such as "a white modern bathtub on a wooden floor," now more detailed descriptions are required, covering the background, ambiance, and surrounding objects.
OpenAI's image generation model, DALL·E 3, has improved data quality by generating more detailed captions. It generates both short and long descriptions, meaning users don’t need to provide lengthy explanations. The AI can create a rough image based on the context, and more detailed images can be generated as specific requirements are provided.
|
|
|
2. Generative Model Structure |
|
|
Once you've built the training data, the next step is the model. The performance of the model can vary significantly depending on how it learns from different parts of the data. A commonly used model for generative tasks is the Diffusion model.
Recently, in generative models, the Diffusion Transformer (DiT) model, which combines the Diffusion model with the Transformer model, has become popular. DiT is a training method that combines the strengths of both models: Diffusion generates high-quality images, while Transformer offers high scalability. In February, DiT was used in OpenAI’s video generation model, Sora, which made a big splash, and since then, many other models have based their work on it.
|
|
|
The Transformer model used in DiT is the ViT (Vision Transformer) architecture. ViT treats an image as a sequence by dividing it into multiple patches, and calculates the relationships between these patches to understand the image's characteristics. One of the key features of DiT is its ability to handle images of various resolutions.
Additionally, it guarantees stable performance even when the patch size is altered or the model size (parameters) is adjusted. It can be considered a generative model structure that incorporates the latest technologies.
|
|
|
3. Method of Training: Rectified Flow |
|
|
Rectified Flow is a recently emerging methodology gaining attention in generative models. Instead of the noise addition and removal process used in Diffusion models, Rectified Flow directly learns the smooth path between data distributions. In other words, it mathematically models the process of transforming data from its initial state to its target state.
The path between data distributions refers to the transformation process where data gradually moves from the initial distribution to the target distribution. The goal is to convert data into a continuous flow, and this path between distributions can be represented using differential equations (ODE) or stochastic differential equations (SDE).
Additionally, the smooth path means that the data transformation process is not abrupt or discontinuous but instead involves gradual, incremental changes at each step. It is designed so that each stage of the transformation logically and naturally connects, bringing the data closer to the target distribution. For example, when restoring a blurry image to high resolution, the process involves sharpening blurred edges, refining contours, and finally filling in textures.
While Diffusion handles this process step-by-step, Flow Matching expresses the entire smooth path as a continuous function, which the model learns. Recently, this approach has been integrated with DiT, moving towards maximizing the performance of the model.
|
|
|
Generative models have made significant progress recently, but there is still much room for improvement. Techniques are needed to maintain consistency in time and motion within videos or to reduce inference time, among other things. Additionally, the perception that the results of creative works are still "AI-like" remains, and there are weaknesses, such as the inability to generate text accurately within images.
Solving these issues seems to be just a matter of time. In just two years, we’ve seen rapid changes, and I’m truly excited to see where the numerous ongoing studies related to generative models will take us in the next two years. 🚀
|
|
|
Your AI Data Standard
Datumo is an all-in-one data platform for your smarter AI
|
|
|
Datumo Inc.
📋 contact@datumo.com
|
|
|
|
|
