Just like how our bodies have the nervous system, electronics have PCB (Printed Circuit Board) inside them. PCB forms the operational circuits of countless electronics that enrich our lives, from refrigerators, smartphones, vacuum cleaners to voice assistants. If the PCB circuit is a house, its foundations would be the green plates, often known as substrates. Just like how strong houses have a strong foundation, there must be no defects in the substrate for the PCB to function effectively.

LG Innotek is a global leader of the substrate and material business. There are largely three sectors of LG Innotek's substrate and material business, which is divided into three product ranges: Package Substrates, where semiconductor chips are upturned, Photomasks, which allows light to shine through in a defined pattern and Tape Substrates, which raises integrated circuits, etc. LG Innotek is the world's leading company with a 32%, 40% and 33% market share in each business sector (based on 2019).

Vision Inspection, the Core Manufacturing Process of Substrate Materials

Vision inspection is checking for defects by inspecting the product with a naked eye with a Machine Vision widely used nowadays, which combines the automatic inspection process using a camera that acts as a human eye. Vision inspection is used in various sectors, from inspecting automobile parts to airport luggages. Some types of vision inspection are important to the point of determining life and death, such as parts inspection for nuclear generators and impurity testing for medicine, etc. Vision inspection also plays a very important role in substrate materials. Because substrates are parts which require millimeter-level precision and determine the lifespan of electronics, not even a small defect is acceptable.

Joint research by researchers from LG AI Research, LG Innotek (Manufacturing Technology Division) and LG Electronics (Production Engineering. Research Institute) was carried out to apply AI technology to defect inspections. The new AI technology, including Self-supervised Representation Learning, were applied to strengthen the standard of inspection. The technology was applied to the flexible substrate material business, which is essential for the manufacturing of smartphones, monitors and TVs, with more than 150 million units are produced by LG Innotek.

Vision inspection is similar to solving a typical supervised classification learning problem. The purpose of this machine learning model is to classify if there is a defect or not by looking at its picture. If the model learns each of the characteristics from the training dataset of labeling whether there is a defect or not, it will be able to distinguish in new photos as well. LG Innotek used these supervised learning models for vision inspection. We then encountered the same problem as countless other AI automation factories that have been operating in the same method. It can be largely classified into three parts.

1. Limited Data of Defects
Because defects do not happen often, there is very little data compared to normal products. Out of millions of products, there are only two to three defects. It is difficult to find and label them to create a dataset for learning.

2. Catastrophic Forgetting
There is no guarantee there will be the same type of defects within the training data in the future. If a new type of defect appears, the AI needs to learn again. In other words, endless maintenance costs occur.

In this case, the method for learning again can be largely classified into two types. Newly educating the AI using a dataset with both old and new data and additionally educating new data from the existing model. The previous method is the most definite method, but there is a waste in resources as large amounts of data must be learned again every time, resulting waste in resources. The latter is less wasteful but previous knowledge may be damaged. Problems arise when the model suddenly forgets previous information, which is called Catastrophic Forgetting.

3. Difficult Reverification
The task for verifying whether each model is working properly is difficult. Because classification results by an AI cannot be completely trusted, it requires a process of verifying its accuracy from extracted sample. Checking whether the product classified as a defect is actually a defect requires long-term concentration.

Is there a method to solve these problems?

Unsupervised Anomaly Detection

It is not just about separately learning the characteristics of normal and abnormal, but using the method of classifying anything that is not normal as abnormal. This is called Anomaly Detection. If the model can learn only normal data, the need to check and label defective data would be unnecessary.

For vision inspections, it can learn visual representation of normal products and classify anything outside the threshold as abnormal. In fact, this method was designed in various ways. However, it was difficult to find in actual manufacturing environments because the accuracy was lower than of the traditional models. Diagram 1 below is a chart that appears in Google Brain Team's 2020 paper ^[1], where all unsupervised learning models (○), including the method presented in the paper (SimCLR), are less accurate than the supervised learning model (X, Supervised on the top left). It can be seen that the number of SimCLR parameters are similar only when it is 16 times that of supervised learning.

Diagram 1 Benchmark results of various unsupervised vision models (ImageNet Top-1 accuracy)^[1]. The unsupervised learning method (SimCLR) presented in the paper is superior to other models, but does not come close to the supervised (top left) model.

In this research for improving vision inspection, researchers implemented a unsupervised method with higher accuracy than supervised methods. The developed model is about to be applied in actual manufacturing lines, which would be the first example of commercialization. It is a method using Contrastive Learning among expressive learning.

The principle for contrastive learning is simple. People know cats are cats and dogs are dogs in pictures. We know whether it is a cat or a dog even if the cat in the picture is upside down or partially taken. The goal of contrastive learning is to teach these human abilities to models.

How can we apply this to unsupervised learning? First, Data Augmentation is carried out from correct data. Transformation, such as cropping, cutting out, coloring, blurring, flipping and rotating, to produce various deformed data. Through appropriate algorithms, data is distinguished between what is seen as normal and what is seen as a defect due to transformations.

Diagram 2 Example of methods of various photo transformations ^[1]

The contrastive learning method learns normal representations from augmented data. Just like how people can accurately know it is a cat even if the photo is partially taken, has a different color or rotated, the learning model will be able to distinguish it as normal even in unfavorable conditions by looking at the picture of the normal product.

The scope of what transformations are normal can vary according to the domain. Researchers decided this category using actual data of substrate materials. More specifically, changing brightness or rotation were included in the category, whereas cutting out some portions and pasting them elsewhere were excluded. It can be seen as following the characteristics of the domain of the substrate material that is the based on the circuit configuration

Diagram 3 Example of converted data of positive samples and negative samples for substrate material(LG AI Research)

After verifying the accuracy of the vision model that learned what is normal through augmented data, it showed better performance than existing supervised learning models. It distinguished normal and defective products with a 110.9% higher accuracy than supervised learning models taught through labeling, which overcame the limitations of unsupervised learning lacking in accuracy even with the benefit of labeling not required. Therefore, the first problem was solved. In the learning process, this type of learning is also called self-directed learning, as the model first assigned a kind of pseudo-label before learning.

Continual learning

How can we solve the second problem that requires effective learning of new defects? Researchers used the Experience Relay Approach among continual learning methods.

The concept is learning new data, as well as important data, selected from previous data. Learning can be carried out effectively with little resources by combining the total of previous and new data. This is a method edited and developed according to substrate material data based on LG AI Research's award winning technology from the CVPR (Computer Vision and Pattern Recognition) 2020 Continual Learning Challenge.

Diagram 4 A chart showing existing data on a two-dimensional plane (LG Innotek, Manufacturing Technology Divison,)

First, an algorithm utilizing Soft Nearest Neighbor Loss was used for selecting important data among previous data. Diagram 4 showed existing data according to normal (blue) and defects (orange, green, red, etc. according to type). If Soft Nearest Neighbor Loss is used, outliers in the boundaries of groups are removed and data with high cohesion, representing the characteristics of the group well, can be selected.

Diagram 5 Conceptual structure of Continual Learning

Additionally, selected data is included in the learning dataset, combined with new data, just like how you would recall only the important events from past memory. A new model is taught with this data. Then, researchers enhanced classification accuracy by applying an Ensemble Model, which combines various models to produce a final prediction result. The classification accuracy for models created through this method even had a 112.5% higher performance, compared to supervised learning. If the new continual learning method is used, we can expect the personnel required for creating a dataset for model updates can be greatly reduced.

eXplainable AI, an AI That Can Explain

No matter what, it is not wise to completely trust the decision of the created model. Confirmation for at least some samples should be carried out to check if the decisions are accurate. However, it is a task which requires extreme attention. How can AI help with this problem? If we can reduce the scope of which area to check for products, it can greatly reduce the labor for vision inspectors. The eXplainable AI (XAI) in the vision sector was the solution to this problem.

LG AI Research introduced a new eXplainable AI vision algorithm at the 2021 AAAI (Association for the Advancement of Artificial Intelligence), SISE(Semantic Input Sampling for Explanation), which greatly exceeded existing methods (Link). Usually, these algorithms find what parts were inspected with importance when a specific model provides a decision for image data and visualizes it as a heat map. However, existing methods had problems. It had low interpretability, was noisy and computational costs were high. SISE narrowly indicates the deciding parts while having low computational costs.

Diagram 6 Comparison of AI explanation algorithms that finds deciding parts that are determined to be defective. The middle picture is the defect. Among the eight algorithms, the bottom right is the heatmap produced by the SISE algorithm, correctly highlighting the defective part (LG Innotek, Manufacturing Technology Division,).

Diagram 6 is the result of applying various explanation algorithms on substrate material image data that is regarded to be defective. Researchers carried out tests such as Energy Based Pointing Game (EBPG) to compare the performances of algorithms. As a result, SISE on the bottom right displayed about three times (298%) more accurate results than the existing leading algorithm of the same type, Grad-CAM (second top left). For reference, in the case of RISE (Randomized Input Sample for Explanation), which was the motif for SISE, it seems to has good explanation ability, but requires a lot of resources that takes up to 10 times the speed compared to SISE.

Smart vision inspection with extreme possibilities

The following is a summary of achievements obtained from applying three new AI technologies.

Technology	Performance	Expected business effect
Unsupervised Anomaly Detection	110.9% enhanced detection accuracy (compared to supervised learning)	80% reduction in data preparation time
Continual Learning	112.5% increase in detection accuracy (compared to supervised learning)	50% reduction in engineer interference
eXplainable AI	298% higher benchmarks (compared to Grad-CAM)	70% reduction in model verification time

Smart Factory received worldwide attention through Germany's pursuance for its Industry 4.0 policy and is seen as one of the most important components for the forth industrial revolution in Korea. Smart Factory is a hyperflexible and self-adapting concept showing manufacturing competence, as it is extremely flexible with the combination of various technology and can independently adjust to change ^[2]. Vision inspection that checks defects in products and AI that takes part in intelligent functions are all technologies indispensable for creating a Smart Factory.

LG researchers recently succeeded in the commercialization of a next-generation vision inspection which has unprecedented autonomy and accuracy by applying cutting-edge AI technology. It can be seen as technology that is very versatile and if it goes through the application process following the characteristics of domain image data, it can be used in countless processes for other products. We hope that the innovation of the manufacturing industry processes brought by AI will lead to the development of more advanced and new products, contributing to the positive virtuous cycle LG AI Research wishes to see.

LG AI Talk Concert | Progress and Performance of Next-Generation Vision Inspection Research – Byungjun Kang Leader (Link)