Denoising Diffusion Probabilistic Models

Super Short Description

  • Paper Link
  • This work deals with score matching. Aim is to generate realistic images (from noise). Two markov chains are used. One is parameter-less which sequentially adds noise to the signal (image). The other is its inverse and it converts a noise to an image sequentially. Deep neural networks are used in the second markov chain. Their novelty is their ability to generate high quality samples from diffusion models. Theoretically, they also show equivalence with denoising score matching with multiple noise levels.

Concise Overview

Definitions

  • Diffusion process: A Markov chain that gradually adds noise to the data until signal is destroyed.
  • Diffusion probablistic model: It is a parameterized markov chain trained which produces samples matching the data after sufficient time. Transitions of this chain uses neural networks. It is also called a reverse process since its aim is to reverse a diffusion process. So, this will start with noise and end with a sample belonging to a data distribution.

Here, the reverse (learnable) process is defined as: drawing

The forward process, a diffusion process with fixed parameters is defined as: drawing

\(x_0\) is the clean image. forward process moves from \(x_{0}\) to \(x_T\), one step at a time and reverse process moves backwards (\(x_T\) to \(x_0\)). ### Reverse process They model the transition probablity using a gaussian distribution \(p_\theta(x_{t-1}|x_t) = N(x_{t-1};\mu_\theta(x_t,t),\sum_{\theta}(x_t,t))\). They fix the covariance matrix to a linear multiple of the Identity matrix. As for the mean term, details are shown in the next section.

Loss

They maximize the log-likelihood of data. They optimize a variational lowerbound of the log-likelihood. drawing On simplification, this boils down to 3 KL Divergence terms drawing

We can use Bayes formula to compute \(q(x_{t-1}\|x_t)\). It turns out that it is also a gaussian distribution. They denote its mean as \(\widetilde{\mu}_t\). Since all terms in KL-divergence are comparison between gaussians, this helps them to compute the divergence more efficient when compared with computing using monte carlo estimates. Additionally, since the parameters of forward process are fixed, \(L_T\) is fixed and therefore can be ignored in training.

\(L_{t-1}\) can be simplified to: drawing

So, simplest choice is that we make our model predict \(\widetilde{\mu}_t\). However, after further simplifing \(L_{t-1}\), another choice arises which essentially directs the model to predict the excess noise present in \(x_t\) with respect to \(x_{t-1}\). This happens primarily because \(q(x_t\|x_0)\) is also gaussian and can be written in a closed form.

Generating process.

To sample \(x_{t-1}\) (from \(p_\theta(x_{t-1}\|x_t)\)), one predicts the excess noise \(\epsilon_\theta(x_t,t)\) and drawing

So, starting from complete noise, one takes backward steps using above formula. At the end, we would have an image belonging to the data distribution.