✨ Creating, Not Classifying

Generation is not creativity in the human sense, but it is more than lookup. A model that can sample convincing images, sentences, music, or code has learned a compressed account of what those objects tend to look like. The central question is how to turn that account into new samples.

The idea

Diffusion: learn to undo corruption

Diffusion models (DALL-E, Stable Diffusion) work by learning to reverse noise:

1. Take a real image
2. Gradually add noise until it's pure static
3. Train a network to predict and remove the noise
4. At generation: start from noise, repeatedly denoise

The denoising network is trained on a supervised task: given a noisy image and a timestep, predict the noise or the clean signal. But to do that well, it must learn the structure of natural images. At generation time, the model starts from noise and repeatedly nudges it toward the learned data distribution.

Autoregressive Generation

Language models generate one token at a time:

1. Given context, predict next token probability
2. Sample a token
3. Add to context
4. Repeat

This factorizes the probability of a whole sequence:

$$p(x_1,\ldots,x_T)=\prod_{t=1}^{T}p(x_t|x_{<t})$$

It is simple, but the conditional distribution can be extremely rich. An essay, a program, or a proof is generated one token at a time, with every new token added to the context for the next prediction.

Latent space and compression

Many generative systems operate in a latent space, a compressed coordinate system where nearby points correspond to related concepts. Stable Diffusion, for example, does not run diffusion directly in pixel space. It uses an autoencoder to move images into a lower-dimensional latent representation, denoises there, then decodes back to pixels. That is much cheaper than modeling every pixel directly.

Latent spaces enable interpolation, editing, style transfer, and semantic directions, but they are not perfectly clean maps of meaning. They are learned spaces shaped by data, architecture, and training objective.

Demo - Diffusion Denoising

Noise t 0.00 Play reverse Draw target Smiley Reset clean data

Key takeaways

• Generative models learn to sample from a data distribution.
• Autoregressive models factor generation into next-token conditionals.
• Diffusion models learn to reverse a gradual noise process.
• Latent-space generation makes high-dimensional sampling cheaper.
• Sampling choices strongly affect diversity, quality, and reliability.

↘ GANs, VAEs, diffusion, and language models are different answers

GANs train a generator against a discriminator in an adversarial game. They can produce sharp samples but can be unstable and suffer mode collapse.

VAEs learn probabilistic latent spaces and optimize an evidence lower bound. They are elegant and stable, but their samples can look blurry if the decoder objective is too simple.

Diffusion models learn a denoising trajectory. They are stable and high quality, but sampling can require many steps.

Autoregressive models generate one discrete token at a time. They are natural for text and code, and increasingly useful in image, audio, and video token spaces.

Math details

Diffusion forward process (adding noise):

$$q(x_t|x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t I)$$

Reverse process (learned denoising):

$$p_\theta(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \sigma_t^2 I)$$

VAE loss (Evidence Lower Bound):

$$\mathcal{L} = \mathbb{E}_{q(z|x)}[\log p(x|z)] - D_{KL}[q(z|x) || p(z)]$$

Work this

Generator choice

Choose a generative modeling approach for text, product images, and molecular candidates. For each, explain whether you care more about likelihood, diversity, controllability, or sample quality.