Authored by Barath Chandran

Published on Oct-12-2025

Diffusion models, first introduced by Sohl-Dickstein et al $^{[1]}$ and later refined by Ho $^{[2]}$, Nichol, and Dhariwal $^{[3]}$, have been a major milestone in visual generation, leading to popular models such as DALL·E, Sora, and Midjourney. But how can we extend this models to generate realisable DNA sequences? Or design novel molecules? Discrete diffusion bridges this gap between image-based diffusion and domains like genomics, materials discovery, and symbolic reasoning. This blog is based on the papers Score Entropy Denoising Diffusion $^{[4]}$ and Discrete Denoising Diffusion Probabilistic $^{[4]}$ models. It attempts to combine both these papers to give a coherent view of how mathematically viable discrete space diffusion models- their training and their inferences- are designed and how they relate to their continuous space counterparts.

So what’s already possible?

Diffusion in continuous space for images and video is now well understood. The fundamental working principle behind these models involves corrupting real data with continuous Gaussian noise during training and progressively denoising it through a learned neural network during inference. In simpler terms, the model learns the score function $^{[6]}$—the gradient of the log-probability of the data—at each time step, and gradually moves towards the data distribution by performing “gradient ascent” across multiple noise levels. You can think of it like climbing uphill on the probability landscape — at each time step, the model nudges the noisy sample a little closer to where the real data manifold is . DDPMs further introduce stochasticity into this process by modeling the denoising probabilistically, where the score/epsilon function provides the mean of the denoised distribution rather than a deterministic prediction.

The sample is iteratively denoised by reversing the gaussian noise.

The sample is iteratively denoised by reversing the gaussian noise.

image.png

So what’s the Problem?

The fundamental problem is that the intuition of continuous diffusion breaks down for discrete data. The Stein score ∇ₓ log p(x) is fundamentally a gradient—a vector in continuous space. In discrete domains (text tokens, categorical labels, molecular graphs), gradients don't exist. So in discrete space

  1. There is no notion of "subtracting" a token or moving a small step toward a category. Instead of modeling a simple denoising direction, we must learn the complete probability distribution $p(xₜ₋₁|xₜ)$ over all possible previous states.
  2. There is no natural sense of direction between categories. In a continuous manifold, the direction A→B can oppose A→D; in discrete space, transitions A→B, A→C, and A→D are all equally valid and unaligned.

A fully connected 5 state discrete space with varying probabilities and the arrows representing their state translations.

A fully connected 5 state discrete space with varying probabilities and the arrows representing their state translations.

A 2D continuous manifold with mixture of 5 gaussians, and the corresponding score function.

A 2D continuous manifold with mixture of 5 gaussians, and the corresponding score function.

So what will you learn?

The intent of the blog is that by combining and finding commonalities in two different papers it would be possible to understand a “standard recipe” for designing discrete-space diffusion methods. To do so, we would be looking at how the three pillars of a generative diffusion model are defined for SEDD and D3PM which in my opinion are the most grounded papers for discrete time and continuous time discrete space diffusion.

  1. Forward Process — how you plan to corrupt the data.

  2. Reverse Process — how you plan to reverse the corruption using a learned model.

  3. Training Objective (Loss) — how you plan to train the model.

The forward process must be designed in a way that a tractable reverse process is possible, the training objective is defined based on what the model predicts during reversal, so in essence there are no diffusion models only “reverse diffusion” models. In the end we see how can customize the corruption process to augment the structure of the data.

Conventions and Notations

The original papers SEDD and D3PM use contradictory conventions in few places for example SEDD considers probability $p$ as a column vector, whereas D3PM as a row vector, which have been resolved here through consistent definitions. For $N$ being the number of discrete states, the various symbols used are: