Projection Regret: Reducing Background Bias for Novelty Detection via Diffusion Models

Current advances in AI models are based on huge amounts of data and models that attempt to encapsulate all knowledge. In contrast, is it possible to build reliable AI models from limited data, such as in manufacturing? Let's say you are training an AI model to detect defects in processed products. Almost all of the learning data will be good behavioral data, and there will be a very limited number of defective data cases. At this point, is there a way to detect defects with a model trained on normal data alone?

This post will introduce the Projection Regret^[1] study conducted by LG AI Research, and also take a look at other good studies related to score-based generative modelling and diffusion models.

1. Projection Regret: Reducing Background Bias for Novelty Detection via Diffusion Models^[1]

Image 1. Projection Regret Abstract

Anomaly detection in the image domain is broadly divided into two categories: discriminative and generative. The first approach is to obtain information to filter out anomaly data using side information learned from image classification^[2]. My research falls into the second category, which involves finding a metric to differentiate normal data from anomalies using a generative model that produces normal data.

Early work in generative model-based anomaly detection used the likelihood of data from explicit probability models, such as VAEs^[3], for detection^[4]. However, these techniques suffered from the issue of assigning higher confidence to outlier data^[5]. Several papers have addressed this by implementing metrics that use the complexity of the image as a criterion^[6,7,8] to further correct for outlier data that is simpler than the original data.

However, there is something that even the current best methods still struggle with: existing models often fail at capturing anomalous data that has a background similar to normal data. This is also related to texture overfitting^[9,10], which occurs in labeled data situations such as classifier robustness and domain generalization. For example, CIFAR-10/CIFAR-100^[11] share a parent dataset. As a result, this makes detection difficult as they display similar background information in natural images, and this background information occupies a large portion of the image.

Diffusion-based Semantic Editing for OOD detection

Image 2. Main idea of the projection regret model^[1]

To solve this case, we proposed a detection scheme using a consistency model^[12] trained on the corresponding normal data. The consistency model has the advantage of one-step distillation of the diffusion model for quick editing, and we chose the consistency model rather than the diffusion model for convenience.

Our observation was that using the diffusion model, we could edit images in such a way that, even without additional label information, by forward diffusing the anomalous data at a reasonable timestep and then recovering the data with reverse diffusion, the background remains similar and the semantics were changed to match normal data (see Image 1 (left)). This is a phenomenon that can be hinted at by existing semantic editing^[13,14]. If we measure the semantic distance between an image and the image recovered by diffusion-based semantic editing, the semantic distance in the abnormal data will be greater than in the normal data. We used LPIPS^[15] as the detection metric.

However, perfect results are not achieved with these metrics alone. The main reason is that the LPIPS metric, which was designed for semantic distance, does not solely focus on semantic information. To solve this problem, we calculated the LPIPS distance between the image generated by additional editing and the transformed image by adding the background information in LPIPS (see Image 1 (right)). By using the difference between the two distance outputs as our final metric, we were able to design a semantic-based metric to find outlier data.

Image 3. Performance difference between diffusion-model-based baseline and projection regret^[1]

Image 4. Performance of projection regret compared to various generative-model-based methods^[1]

Image 5. Results of the proposed U-Net-based distance metric^[1]

The metric we designed not only outperforms diffusion-model-based algorithms in various image domains but also outperforms unsupervised-model-based anomaly detection techniques (see Images 2 and 3). We also designed a metric extracted from the U-Net of diffusion models as an alternative to LPIPS, and it also performs comparably to LPIPS.

Next, we will look at two key papers related to diffusion models.

2. Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation^[19]

Image 6. Different weighting^[19] used by current diffusion models

First, we will introduce “Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation” presented by Google DeepMind at NeuroIPS 2023. This paper discussed weighting settings that combine training loss in diffusion models with different noise levels.

It proves that monotonic weighting can be represented by ELBO when data is augmented with a Gaussian Perturbation and designs a monotonic form of weighting for the EDM objective. The authors applied this weighting to the ImageNet dataset at various resolutions and achieved consistent and significant performance gains over traditional scheduling.

3. Text-to-Image Diffusion Models are Zero-Shot Classifiers^[20]

Image 7. Zero-Shot Classification process^[20]

Similarly, Google DeepMind’s “Text-to-Image Diffusion Models are Zero-Shot Classifiers” utilizes a text-to-image diffusion model and shows the results of applying it to a variety of downstream tasks in addition to existing generation.

In this paper, prompts related to each class are given as conditions and each result is generated as guidance. Then, zero-shot classification is performed by providing the difference from the existing model as a score and summing it up at several timesteps, using the class with the minimum difference as the output. This method showed better performance than the zero-shot classification of models, such as existing CLIP, and showed slightly more robust results even with texture bias, which is prone to classifier overfitting. Although it is based on a diffusion model and requires a lot of evaluation time, it looks promising.

Future Plans

The Advanced ML Lab is working on a paper in collaboration with other labs on out-of-distribution detection. Briefly, our research combines image editing technologies, such as DiffEdit and SDEdit, with the aforementioned concepts to generate effective outlier data that improves anomaly detection performance. Unlike the baseline projection regret we used, the stable diffusion model shows better performance.

Additionally, with the increasing attention on AI-generated images, detecting these images^[17,18] is becoming necessary. We are also conducting research on the safe use of such AI-generated images. Finally, we are also interested in expanding our research to various fields, including time-series data.

LG AI Research has presented research at various conferences such as NeurIPS and KCCV, and is actively engaged in technical exchanges with many researchers. Going forward, LG AI Research will continue to strengthen collaboration in various research fields and contribute to increasing the reliability and scalability of AI technology. We will continue to research and challenge ourselves to make AI a safer and more reliable technology.