[NeurIPS 2021] Part 1: Generative model - Diffusion model Review

The Conference and Workshop on Neural Information Processing Systems(NeurIPS 2021) is a machine learning and computational neuroscience conference founded in 1987. It is also one of the most influential conferences in the field of AI and machine learning with the average acceptance rate of 20%. Here is a summary of the research papers shared at NeurIPS 2021, picked by three researchers from Applied AI Research lab, LG AI Research. - Part 1: Generative model - Diffusion model Review - Hyemin Jung from Applied AI Research lab - Part 2: Variational Deep Embedding-Based Deep Clustering Review - Jinseok Yang from Applied AI Research lab - Part 3: Neural Architecture Search Review - Wonbin Ahn from Applied AI Research lab

Introduction

At the LG AI Research’s Applied AI Research Lab, generative models are being vigorously studied. Generally speaking, a deep learning model can be divided into two categories: discriminative and generative. Applied AI Research Lab is researching anomaly detection, representation learning, music generation, recommendation, etc. in time-series data.

The topic of this blog post touches on the subject of various generative models as well as the diffusion model, one of the most studied generative models recently. This blog will explain representative papers on diffusion model and variational diffusion model published in the NeurIPS 2021 based on said paper.

Generative model

What is a generative model?
A generative model is a model that estimates the sampled distribution of given data x. What it means for a deep learning model to estimate the distribution is that the model’s output may be the parameter for determining the distribution or the marginal likelihood of data output. A generative model can be used to determine the likelihood of test data and generate new data.

The performance of a generative model is shaped by how well it predicts the likelihood p(x) value of given data x and the quality of data x generated.

Comparison of generative models
There are 6 types of generative models. Each of their characteristics, as well as advantage and disadvantage are as follows:

Auto-regressive models(ARMs)
An ARM is the product of conditional probability of sequential variables and used to calculate the likelihood of data. Its advantage is that it can determine the likelihood of a given data and its disadvantage is its slow sampling and inability to study latent features in the data.

Variational Auto encoders(VAEs)
A VAE is a form of latent variable-based generative model that determines joint distributions of data x and latent variable z, then marginalizes data x. Its advantage is fast learning and sampling, and that the latent features of the data can be learned, while its disadvantages are intractable likelihoods and restrictive prior distributions.

Energy Based Models(EBMs)
An EBM estimates distribution using energy functions. It is simple with no constraints, but its disadvantage includes intractable likelihood and sampling.

Generative Adversarial Networks(GANs)
A GAN is a type of learning framework where the discriminator and generator contest with each other in adversarial directions and learn to generate new data. Its advantages include good sample quality, as well as fast training and sampling. Its disadvantages include undefined likelihood and unstable training.

Normalizing Flows
A normalizing flow model estimates distribution by using invertible mapping functions to transform a simple distribution, p(z), into a complex one, p(x). It can determine exact likelihoods and enables fast sampling, but its disadvantage is the limiting aspects on architectures.

Diffusion
A diffusion model estimates distribution by gradually adding noise to data x and generating noise data, then learning the process of noise data returning to data x. Although likelihoods are intractable, it has good likelihood values and can estimate complex distributions well without limitations.

Of the various generative models above, diffusion models are being heavily studied the most recently. Many diffusion models, which have better likelihood performance than autoregressive-based generative models, and generate higher quality samples than GAN-based models were published in the NeurIPS 2021.

Now, I will give a basic description of diffusion models and explain the ‘variational diffusion model,’ the diffusion model published in NeurIPS 2021.

What is a diffusion model?

Diffusion probabilistic models
The first ever paper that put forth a diffusion model was “Deep Unsupervised Learning using Nonequilibrium Thermodynamics” (2015) ^[1]. The idea of a diffusion model came from thermodynamics, and it can be divided into two stages. First, noise is gradually added to given data x. This process is called the “Diffusion process.” Second, the reverse process of the diffusion process defined earlier is calculated. In this process, the noise is gradually removed from the noise data transforming it into original data x. A deep learning model is used to learn this reverse diffusion process, and this process is called “Reverse diffusion process.”

1.Diffusion process

In a diffusion process, noise is added to given data x at every time step. At step 0, noise is added to move to step T, and the latent variable z_t is the noise added at each step. Mathematically expressing this diffusion process gives us the Gaussian distribution as shown below: 

$q(z_t\overline{)z_{t-1}) = N(\sqrt{1-{\beta }_t}{z}_{t-1}},{\beta }_{t}I)$

In the equation above, the signal, $\sqrt{1-{\beta }_t}$ is taken from the previous step, latent variable $z_{t-1}$ and noise, ${\beta }_t$, is added to determine $z_t$. Here, ${\beta }_t$ is the quantity of noise set by the user. As the time step gets bigger, a larger value is assigned to the noise. ${\beta }_0<{\beta }_1<\cdots <{\beta }_T$
Once noise is added all the way to step T, z_T will eventually become an Isotropic Gaussian distribution.

A diffusion process assumes a Markov attributes. Therefore, once data x is given as a product of distribution, as shown below, the latent variable distribution can be extracted.

$q\left(z_{1:t}\overline{)z_0}\right)=\prod _{t=1}^{T}q\left(z_t\overline{)z_{t-1}}\right)$

When data $\mathrm{x}\left(=z_0\right)$ is given, the distribution of latent variable $z_t$ is the Gaussian distribution.

$q\left(z_t\overline{)z_0}\right)=N\left(z_t;\sqrt{\overline{{\alpha }_t}}{z}_0,\left(1-{\overline{\alpha }}_t\right)I\right), {\alpha }_t=1-{\beta }_t {\overline{\alpha }}_t=\prod _{i=1}^T{\alpha }_i$

The equation above signifies specific noises added with $\mathrm{x}\left(=z_0\right)$ as basis.

When $z_t, x\left(=z_0\right)$ are given as below using the Gaussian distributions defined above, we can deduce the Gaussian distribution of $z_{t-1}$. 

$q\left(z_{t-1}\overline{)z_t,z_0}\right)=N\left(z_{t-1};{\tilde{\mu }}_t\left(z_t,z_0\right),{\tilde{\beta }}_{t}I\right)$
$\tilde{\mu }\left(z_t,z_0\right)=\frac{1}{\sqrt{{\alpha }_t}}\left(z_t-\frac{{\beta }_t}{\sqrt{1-{\overline{\alpha }}_t}}{\in }_t\right), {\tilde{\beta }}_t\left(z_t,z_0\right)=\frac{1-{\overline{\alpha }}_{t-1}}{1-{\alpha }_t}{\beta }_t$

_{${\in }_t~N\left(0,I\right)$} is the value used to determine $z_t~N\left(z_t;\sqrt{{\overline{\alpha }}_t}{z}_0,\left(1-{\overline{\alpha }}_t\right)I\right)$ using the reparameterization trick. This equation signifies the latent variable distribution of the previous time step, and this is important as it will serve as the solution when the model learns the reverse diffusion process.

To sum it up, a diffusion process is a course of adding noise ratio value, ${\beta }_t$, at every time step and all relevant distributions can be expressed as Gaussian distributions. No learning takes place in this process.

2. Reverse Diffusion process

The reverse process of the former diffusion process is a generative model where the noise data return back to the original data x. Although a diffusion process is deliberately defined as a Gaussian distribution, distributions portraying the reverse diffusion process are still unknown. We can learn about unknown reverse diffusion processes through the model.

A reverse diffusion process can be expressed as the equation below:

$p_{\theta }\left(z_{0:T}\right)=p\left(z_T\right)\prod _{t=1}^T{p}_{\theta }\left(z_{t-1}\overline{)z_t}\right)$

$p_{\theta }\left(z_{t-1}\overline{)z_t}\right)$ is the reverse distribution estimated by the model. This distribution is a Gaussian distribution and the model learns the average, one of parameters of the Gaussian distribution. The equation is as shown below:

$p_{\theta }\left(z_{t-1}\overline{)z_t}\right)=N\left(z_{t-1};{\mu }_{\theta }\left(z_t,t\right),{\Sigma }_{\theta }\left(z_t,t\right)\right)$

3. Loss
Let’s review the reverse diffusion process, that is, the loss equation to learn the generative model. The purpose of a generative model is distribution estimate, which can be determined by calculating the likelihood of given data x. Therefore, an objective equation must represent log likelihood of x. The loss equation can be deduced from this objective equation. The detailed process of deduction will not be mentioned here.


A loss equation is also called Variational Lower Bound (VLB) and the meaning of each term is as follows:

Because term $D_{KL}\left(q\left(z_T\overline{)x}\right)\parallel p_{\theta }\left(z_T\right)\right)$ combines two Gaussian distributions and both $q\left(z_T\overline{)x}\right),p_{\theta }\left(z_T\right)$ are Gaussian distributions, they can be calculated in closed form.

Term $log\left(p_{\theta }\left(x\overline{)z_1}\right)\right)$ reconstructs the original x in latent variable z₁ and this term can also be calculated directly.
Both terms above can be calculated whether the models learned it or not.

Term $\sum _{t=1}^T{D}_{KL}\left(q\left(z_{t-1}\overline{)z_t,x}\right)\parallel p_{\theta }\left(z_{t-1}\overline{)z_t}\right)\right)$ is called a diffusion loss and it enables the learning of the reverse diffusion process. $\left(x=z_0\right)$

$q\left(z_{t-1}\overline{)z_t,z_0}\right)=N\left(z_{t-1};{\tilde{\mu }}_t\left(z_t,z_0\right),{\tilde{\beta }}_{t}I\right)$
${\tilde{\mu }}_t\left(z_t,z_0\right)=\frac{1}{\sqrt{{\alpha }_t}}\left(z_t-\frac{{\beta }_t}{\sqrt{1-{\overline{\alpha }}_t}}{\in }_t\right), {\tilde{\beta }}_t\left(z_t,z_0\right)=\frac{1-{\overline{\alpha }}_{t-1}}{1-{\overline{\alpha }}_t}{\beta }_t$

is the solution distribution, and it is learned to bring the model distribution ($p_{\theta }\left(z_{t-1}\right|z_t)=N(z_{t-1};{\mu }_{\theta }(z_t,t),{\Sigma }_{\theta }(z_t,t))$ closer to $q\left(z_{t-1}\overline{)z_t,z_0}\right)$ distribution. Both distributions are Gaussian distribution and KL divergence is calculated as shown below: 

$D_{KL}\left(q\left(z_{t-1}{\overline{)z}}_t,x\right)\parallel p_{\theta }\left(z_{t-1}\overline{)z_t}\right)\right)=E_{x,z}\left[\frac{1}{2{||{\Sigma }_{\theta }\left(z_t,t\right)||}_2^2}{||{\tilde{\mu }}_t\left(z_t,x\right)-{\mu }_{\theta }\left(z_t,t\right)||}^2\right]$

We have briefly gone over the concept of diffusion model and loss equations. A wide range of diffusion models have been put forth based on the information above, and many papers have been published in the NeurIPS 2021. Of the models above, I will explain the variational diffusion model.

Variational diffusion model^[2]

Motivation
There have been many Diffusion models put forth with adequate performances that are based on the generated image quality, but Diffusion models based on likelihood performed poorly compared to autoregressive-based generative models. This paper proposes diffusion models that have overcome such limitations and displayed SOTA performance based on likelihood.


Contribution
The contributions of this paper are as follows:

- Use of Fourier feature is proposed to improve likelihood performance.
- Added noise schedule during diffusion process are determined based on how the model learns.
- Proposed a simpler form of VLB.
- Theoretically proved that the higher the number of steps is in the diffusion process, the better the performance improves.
- Deduced diffusion loss from infinite time steps (continuous time steps).
- Theoretically illustrated that the relationship between the loss equation from continuous time steps and existing diffusion models.

Forward time Diffusion process


The first distribution in the diffusion process as proposed in this paper is as follows:

$q\left(z_t\overline{)x}\right)=N\left({\alpha }_{t}x,{\sigma }_t^{2}I\right)$

When data x is given, the distribution of latent variable $z_t$ will be a Gaussian. ${\alpha }_t$ is the quantity of signal used from the original data x, and $z_t$ can be calculated after adding the noise of ${\sigma }_t^2$. α_t is referred to as signal and ${\sigma }_t^2$ is noise. The relationship between ${\alpha }_t$ and ${\sigma }_t^2$ is as shown below:

${\alpha }_t=\sqrt{1-{\sigma }_t^2}$

In this paper, a new parameter SNR(t) was proposed, which indicates signal to ratio at time t.

$\mathrm{SNR}\left(\mathrm{t}\right)=\frac{{\alpha }_t^2}{{\sigma }_t^2}$

Greater SNR(t) value signifies more signals than noises and smaller SNR(t) value signifies more noises than signals. Therefore, SNR(t) value shrinks as t gets bigger, that is, as the diffusion process moves forward.


In addition, this paper proposes that noise schedule is not predetermined but learned by the models.

${\sigma }_t^2=sigmoid\left({\gamma }_{\eta }\left(t\right)\right)$

${\gamma }_{\eta }\left(t\right)$ is a monotonic neural network model, which gives higher outputs as input, t, grows. A monotonic neural network was chosen because a diffusion process needs its data to be noisy and the noise value, ${\sigma }_t^2$, must grow. In this paper, three linear layers were used for ${\gamma }_{\eta }\left(t\right)$.

Therefore, ${\alpha }_t, {\sigma }_t^2, \mathrm{SNR}\left(\mathrm{t}\right)$ can be expressed as below:

${\sigma }_t^2=sigmoid\left({\gamma }_{\eta }\left(t\right)\right)=\frac{\mathrm{SNR}\left(\mathrm{t}\right)}{1+\mathrm{SNR}\left(\mathrm{t}\right)}$
${\alpha }_t^2=sigmoid\left(-{\gamma }_{\eta }\left(t\right)\right)=\frac{1}{1+\mathrm{SNR}\left(\mathrm{t}\right)}$
$\mathrm{SNR}\left(\mathrm{t}\right)=\frac{sigmoid\left(-{\gamma }_{\eta }\left(t\right)\right)}{sigmoid\left({\gamma }_{\eta }\left(t\right)\right)}=exp\left(-{\gamma }_{\eta }\left(t\right)\right)$

(SNR(t) is an invertible monotonically decrease function)

The distribution comprising the diffusion process is as shown below: (0<s<t<1)

$q\left(z_t\overline{)z_s}\right)=N\left({\alpha }_{t\overline{)s}}{z}_s,{\sigma }_{t\overline{)s}}^{2}I\right)$
${\alpha }_{t\overline{)s}}=\frac{{\alpha }_t}{{\alpha }_s}, {\sigma }_{t\overline{)s}}^2={\sigma }_t^2-{\alpha }_{t\overline{)s}}^2{\sigma }_s^2$

The equation signifies that, when the latent variable of the previous step is given, the distribution of the next latent variable will be a Gaussian, and that schedule information was taken at $z_s$ and noise was added at $z_t$. The process which determines ${\alpha }_{t\overline{)s}}, {\sigma }_{t\overline{)s}}^2$ can also be deduced but it is omitted here.

Using the Gaussian distributions defined above, when given z_t,x, the following Gaussian distribution can be deduced for the previous step, $z_s$, as shown below:

$q\left(z_s\overline{)z_t,x}\right)=N\left(z_s;{\mu }_Q\left(z_t,x;s,t\right),{\sigma }_Q^2\left(s,t\right)I\right)$
${\mu }_Q\left(z_t,x;s,t\right)=\frac{{\alpha }_{t\overline{)s}}{\sigma }_s^2}{{\sigma }_t^2}{z}_t+\frac{{\alpha }_s{\sigma }_{t\overline{)s}}^2}{{\sigma }_t^2}x, {\sigma }_Q^2\left(s,t\right)={\sigma }_{t\overline{)s}}^2\frac{{\sigma }_s^2}{{\sigma }_t^2}$

Reverse time Generative model


In the reverse diffusion process, original data x is generated by determining the latent variables in T=1 → T=0 direction. This process can be seen as a hierarchical generative model. There are two cases: 1) discrete time case, where there are discrete time steps with finite number of Ts between T=1 and T=0; and 2) continuous time case, where there are infinite time steps (T → ∞) between T=1 and T=0. First, discrete time case will be explained here.

Discrete time case
The span between 0 and 1 was uniformly discretized to comprise T number of time steps.


In this case, the generative model for x can be described by the following equation:

$p\left(x\right)={\int }_{z}p\left(z_1\right)p\left(x\overline{)z_0}\right)\prod _{i=1}^{T}p\left(z_{s\left(i\right)}\overline{)z_{t\left(i\right)}}\right)$

The equation above denotes that starting from noise, $z_1$, and given $z_{t\left(i\right)}$, the outcome is $z_{s\left(i\right)}$, and at $z_0$ the model begins to return to original data x. The meaning of each term is as explicated below:

Because $q\left(z_1\overline{)x}\right)=N\left(z_1;0,I\right)$, $p\left(z_1\right)=N\left(z_1;0,I\right)$ is also defined as a Gaussian.
In the equation $q\left(x\overline{)z_0}\right)=\frac{q\left(z_0\overline{)x}\right)q\left(x\right)}{q\left(z_0\right)}$, at SNR(0), the value of $q\left(z_0\overline{)x}\right)$ is dominated by the greater value of $q\left(z_0\overline{)x}\right)$ as the noise value is small, rendering z₀ and x analogous. $q\left(z_0\overline{)x}\right)=N\left({\alpha }_{0}x,{\sigma }_0^{2}I\right)$, and element-wise scale/shift takes place when determining z₀ at x, the other dimensions are not affected. Therefore, it can be defined that $p\left(x\overline{)z_0}\right)=\prod _{i}p\left(x_i\overline{)z_{0,i}}\right)$.

$p\left(z_{s\left(i\right)}\overline{)z_{t\left(i\right)}}\right)$ is a reverse distribution, which the model must learn. In other words,
It must learn to approach $q\left(z_s\overline{)z_t,x}\right)=N\left(z_s;{\mu }_Q\left(z_t,x;s,t\right),{\sigma }_Q^2\left(s,t\right)I\right)$,  ${\mu }_Q\left(z_t,x;s,t\right)=\frac{{\alpha }_{t\overline{)s}}{\sigma }_s^2}{{\sigma }_t^2}{z}_t+\frac{{\alpha }_s{\sigma }_{t\overline{)s}}^2}{{\sigma }_t^2}x$, ${\sigma }_Q^2\left(s,t\right)={\sigma }_{t\overline{)s}}^2\frac{{\sigma }_s^2}{{\sigma }_t^2}$, distributions. $p\left(z_{s\left(i\right)}\overline{)z_{t\left(i\right)}}\right)$ distribution can be deduced in two viewpoints. First, from the viewpoint of denoising model. (Here, s(i) is s, t(i) is t.)

$p\left(z_s\right|z_t)=N(z_s;{\mu }_{\theta }(z_t;s,t),{\sigma }_Q^2(s,t)I)$
${\mu }_{\theta }(z_t;s,t)=\frac{{\alpha }_{t|s}{\sigma }_s^2}{{\sigma }_t^2}{z}_t+\frac{{\alpha }_s{\sigma }_{t|s}^2}{{\sigma }_t^2}{\hat{\mathbf{x}}}_{\mathbf{\theta }}\mathbf{(}{\mathit{z}}_{\mathbf{t}}\mathbf{;}\mathit{t}\mathbf{)}, {\sigma }_Q^2(s,t)={\sigma }_{t|s}^2\frac{{\sigma }_s^2}{{\sigma }_t^2}$

${\hat{\mathrm{x}}}_{\mathrm{\theta }}(z_{\mathrm{t}};t)$ is a denoising model and predicts original data x once z_t is entered. 

Second, from the viewpoint of a noise prediction model; in the equation $q(z_t│x)=N({\alpha }_{t}x,{\sigma }_t^{2}I)$, when z_t is calculated using the reparameterization trick, we get $z_t={\alpha }_{t}x+{\sigma }_t\odot \in$, $\in ~N(0,I)$. This equation can be reformulated in term of x, that is, $x=\frac{z_t-{\sigma }_t\odot \in }{{\alpha }_t}$, where ${\hat{x}}_{\theta }\left(z_t;t\right)$ can be described as shown below:

${\hat{x}}_{\theta }\left(z_t;t\right)=\frac{z_t-{\sigma }_t\odot {\hat{\in }}_{\theta }(z_t;t)}{{\alpha }_t}$

${\hat{\in }}_{\theta }\left(z_t;t\right)$ is a noise prediction model, where inputting z_t predicts $\in$. In this viewpoint, $p\left(z_s\overline{)z_t}\right)$ is as shown below:

$p\left(z_s\overline{)z_t}\right)=N\left(z_s;{\mu }_{\theta }\left(z_t;s,t\right),{\sigma }_Q^2\left(s,t\right)I\right)$
${\mu }_{\theta }\left(z_t;s,t\right)=\frac{1}{{\alpha }_{t\overline{)s}}}{z}_t-\frac{{\sigma }_{t\overline{)s}}^2}{{\alpha }_{t\overline{)s}}{\sigma }_t}{\hat{\mathbf{\in }}}_{\mathbf{\theta }}\left(z_t;t\right), {\sigma }_Q^2\left(s,t\right)={\sigma }_{t\overline{)s}}^2\frac{{\sigma }_s^2}{{\sigma }_t^2}$

Denoising model viewpoint is intuitively easy to understand, but noise prediction viewpoint was applied in experiments.

Therefore, the generative model can be rearranged as shown below:

$p\left(x\right)={\int }_{z}p\left(z_1\right)p\left(x\overline{)z_0}\right)\prod _{i=1}^{T}p\left(z_{s\left(i\right)}\overline{)z_{t\left(i\right)}}\right)$
$p\left(z_1\right)=N\left(z_1;0,I\right)$
$p\left(x\overline{)z_0}\right)=\prod _{i}p\left(x_i\overline{)z_{0,i}}\right)$
$p\left(z_s\overline{)z_t}\right)=N\left(z_s;{\mu }_{\theta }\left(z_t;s,t\right),{\sigma }_Q^2\left(s,t\right)I\right), {\mu }_{\theta }\left(z_t;s,t\right)=\frac{{\alpha }_{t\overline{)s}}{\sigma }_s^2}{{\sigma }_t^2}{z}_t+\frac{{\alpha }_s{\sigma }_{t\overline{)s}}^2}{{\sigma }_t^2}{\hat{x}}_{\theta }\left(z_t;t\right), {\sigma }_Q^2\left(s,t\right)={\sigma }_{t\overline{)s}}^2\frac{{\sigma }_s^2}{{\sigma }_t^2}$

Or

$p\left(z_s\overline{)z_t}\right)=N\left(z_s;{\mu }_{\theta }\left(z_t;s,t\right),{\sigma }_Q^2\left(s,t\right)I\right), {\mu }_{\theta }\left(z_t;s,t\right)=\frac{1}{{\alpha }_{t\overline{)s}}}{z}_t-\frac{{\sigma }_{t\overline{)s}}^2}{{\alpha }_{t\overline{)s}}{\sigma }_t}\hat{\in }\left(z_t;t\right), {\sigma }_Q^2\left(s,t\right)={\sigma }_{t\overline{)s}}^2\frac{{\sigma }_s^2}{{\sigma }_t^2}$

Loss equation
The loss equation for learning the above generative model is as shown below:


Prior loss and reconstruction loss can be determined using the reparameterization trick. Diffusion loss is the term that can be reduced through model learning. In discrete time cases, diffusion loss equation is as shown below:


It opts to close the gap between reverse distribution of the solution and model distribution. The equation deduced from the above equation ($p\left(z_s\right|z_t)=N(z_s;{\mu }_{\theta }(z_t;s,t),{\sigma }_Q^2(s,t)I)$, ${\mu }_{\theta }(z_t;s,t)=\frac{{\alpha }_{t|s}{\sigma }_s^2}{{\sigma }_t^2}{z}_t+\frac{{\alpha }_s{\sigma }_{t|s}^2}{{\sigma }_t^2}{\hat{x}}_{\theta }(z_t;t)$, ${\sigma }_Q^2\left(s,t\right)={\sigma }_{t\overline{)s}}^2\frac{{\sigma }_s^2}{{\sigma }_t^2}$) in denoising viewpoint is as shown below:


This equation determined the difference between the actual solution x and the output after inputting latent variable z_t at each step, and then weight was multiplied to the answer. A penalty was given in proportion to the difference of signal to ratio values between current and previous steps. In other words, greater penalty was given to more noisy data.

In noise prediction viewpoint, if ($p\left(z_s\right|z_t)=N(z_s;{\mu }_{\theta }(z_t;s,t),{\sigma }_Q^2(s,t)I)$, ${\mu }_{\theta }(z_t;s,t)=\frac{1}{{\alpha }_{t|s}}{z}_t-\frac{{\sigma }_{t|s}^2}{{\alpha }_{t|s}{\sigma }_t}{\hat{\in }}_{\theta }(z_t;t)$, ${\sigma }_Q^2(s,t)={\sigma }_{t|s}^2\frac{{\sigma }_s^2}{{\sigma }_t^2}$) are expressed as SNR(t) and, $z_{t\left(i\right)}$ is calculated using the reparameterization trick, a simple equation can be deduced, as shown below:


Continuous time case
This paper mathematically proved that diffusion loss decreased as the number of time steps increased. As such, a diffusion loss equation was deduced for the case of infinite time steps (T → ∞). In this case, t is a continuous value. A limit can be applied to the loss equation of discrete case diffusion and deduce the following equation:


It is apparent that the weight term has been replaced by a differential value.
The equation deduced from the denoising viewpoint is as shown below: $\left(\mathrm{SNR}\left(\mathrm{t}\right)=exp\left(-{\gamma }_{\eta }\left(t\right)\right)\right)$


As mentioned earlier, a noise schedule must be learned, and therefore, the invertible monotonically decrease function, SNR(t), must also be learned. $\left(\mathrm{SNR}\left(\mathrm{t}\right)=exp\left(-{\gamma }_{\eta }\left(t\right)\right)\right)$. This paper revealed through proof that in a continuous case, SNR(t) did not influence the diffusion loss equation; only SNR(0) and SNR(1) values influenced the diffusion loss. Thus, it is not necessary to learn SNR(t) parameter η to reduce the diffusion loss. Only noise prediction model parameter θ, as well as SNR(0) and SNR(1) values need to be learned.

Model parameters can be learned by calculating the equation above using the Monte Carlo (MC) estimator. The MC estimator equation is as shown below:


This paper proposes to learn parameter η of $\mathrm{SNR}\left(\mathrm{t}\right)=exp\left(-{\gamma }_{\eta }\left(t\right)\right)$ in order to reduce dispersion of the MC estimator.

Fourier Features
In summary, the generative model is as follows:

$p\left(x\right)={\int }_{z}p\left(z_1\right)p\left(x\right|z_0\left)\prod _{i=1}^{T}p\right(z_{s\left(i\right)}\left|z_{t\left(i\right)}\right)$
$p\left(z_1\right)=N(z_1;0,I)$
$p\left(x\right|z_0)={\prod }_{i}p(x_i\left|z_{0,i}\right)$
$p\left(z_s\right|z_t)=N(z_s;{\mu }_{\theta }(z_t;s,t),{\sigma }_Q^2(s,t)I), {\mu }_{\theta }(z_t;s,t)=\frac{{\alpha }_{t|s}{\sigma }_s^2}{{\sigma }_t^2}{z}_t+\frac{{\alpha }_s{\sigma }_{t|s}^2}{{\sigma }_t^2}{\hat{x}}_{\theta }(z_t;t), {\sigma }_Q^2(s,t)={\sigma }_{t|s}^2\frac{{\sigma }_s^2}{{\sigma }_t^2}$

Or

$p\left(z_s\right|z_t)=N(z_s;{\mu }_{\theta }(z_t;s,t),{\sigma }_Q^2(s,t)I), {\mu }_{\theta }(z_t;s,t)=\frac{1}{{\alpha }_{t|s}}{z}_t-\frac{{\sigma }_{t|s}^2}{{\alpha }_{t|s}{\sigma }_t}{\hat{\in }}_{\theta }(z_t;t), {\sigma }_Q^2(s,t)={\sigma }_{t|s}^2\frac{{\sigma }_s^2}{{\sigma }_t^2}$

There are two cases, namely discrete time and continuous time, of diffusion loss equation when learning this. Diffusion loss of discrete time case


Diffusion loss of continuous time case (MC estimator).


In other words, the model must learn how to enter latent variable z_t at each time step to predict $\in$. In this paper, in order to improve the likelihood performance, it was proposed to calculate high frequency Fourier feature of z_t using the equation below


and concatenate to z_t, then proceed with input. The input values for high frequency Fourier feature values have been amplified in various scales, and adding them properly captured the fine scale details.

Experiment
The VDM proposed in this paper displayed SOTA record in the likelihood benchmark compared to other generative models.


Generated image quality is as shown below:


Conclusion

I have set my goal to estimate given data distributions and resolve the relevant downstream tasks. I am currently researching various generative models of images, sequential data and graph data. When writing this blog, I was able to identify the advantages of diffusion models, a type of generative model. However, the likelihood value of a diffusion model is, at the end, the lower bound value of logp(x) and a gap still exists between the two. Moving forward, I believe we need to contemplate on how to reduce this gap.