From Labeling to Deployment: VLM and Agentic AI-Based Autonomous Vision Inspection

With the advance of smart factories, deep learning-based vision inspection systems have become an essential solution on manufacturing floors. However, even after system deployment, engineers continuously face a structural limitation: the manual labor and operational burden associated with retraining and redeploying models.

Whenever Concept Drift occurs due to process variations, lighting changes, or the appearance of new defect types on the production line, the reliability of existing inspection models degrades^[1]. To mitigate this, Human-in-the-Loop (HITL) frameworks have been adopted to algorithmically extract 'Uncertainty Samples'—data that the model struggles to classify—in an active learning manner^[2]. Yet, the actual bottleneck arises in the subsequent steps.

Interpreting the visual context of selected uncertainty data, assigning accurate labels, feeding them into the training pipeline, and finally deploying the updated model remains heavily dependent on the experience and intuition of skilled engineers. Sorting through tens of thousands of images to manually isolate Edge Cases and label them is not only time-consuming and costly, but it also suffers from inconsistencies driven by subjective human judgment^[3]. Consequently, human intervention acts as a severe bottleneck across the entire operational pipeline—from data refinement to retraining and deployment.

To overcome these constraints, modern manufacturing floors demand a paradigm shift: moving beyond simple script-based automation toward an 'Agentic System equipped with autonomous reasoning and execution capabilities'^[4].

This article presents the architecture and potential of an autonomous vision inspection framework designed to virtually eliminate human intervention (Zero-Human Intervention). We leverage Vision-Language Models (VLMs) capable of high-level visual context reasoning to replace human intuition in refining and labeling uncertainty data^[5], alongside Agentic AI technology to achieve seamless Tool Orchestration.

Figure 1. From Uncertainty Extraction to Autonomous Deployment: A Zero-Human Vision Inspection Framework

Figure 1 illustrates the overall operational flow of the proposed framework. While there have been ongoing efforts to establish autonomous operations in manufacturing vision inspection, conventional pipeline architectures (As-Is) carry fundamental limitations. When ambiguous or unlabeled data occurs during line operations, selecting and assigning ground truth labels relies entirely on manual human judgment. The entire workflow—spanning uncertainty sample selection, database cleansing, model retraining, simulation verification, and field redeployment—causes a severe human bottleneck, hindering rapid redeployment, which is the cornerstone of autonomous operations.

The LLM Agent-based autonomous operational framework proposed in this study fundamentally resolves these structural limitations. As shown in Figure 2, the LLM-based AI Agent governs the entire workflow, achieving a 100% autonomous operational environment that seamlessly connects and executes all processes from data preparation to model deployment without human intervention. While the framework primarily operates in a fully autonomous mode, it also offers an optional interactive copilot mode, allowing engineers to issue natural language commands to collaborate when necessary.

Figure 2. LLM Agent Conceptual Diagram

This agent framework operates through four core autonomous mechanisms:

First is the Test-Time Uncertainty Estimation and Auto-Sampling stage. As the field inspection model performs inference, it identifies 'Uncertainty Samples' that fall near the boundary between OK and NG in real time and automatically transfers them to the labeling database.

Second is VLM-based Multimodal Auto-Labeling. Upon receiving the collected uncertainty samples alongside product and inspection domain prompts, the Vision-Language Model (VLM) performs comparative reasoning against normal reference images on a grid-area basis.

Third, Noise-Robust Learning Policy (Learning with Noisy Labels). To account for potential minor errors in the pseudo-labels generated by the VLM, a similarity-based learning policy operating on visual feature embeddings is applied. This ensures stable model retraining without being degraded by label noise in the collected data.

Finally, Agentic AI-based Autonomous Orchestration and Feedback Loop-based Deployment. The AI Agent autonomously reasons and executes the entire pipeline—from extracting uncertainty data and performing VLM auto-labeling to storing training datasets, triggering retraining pipelines, conducting simulation verification, and deploying the final model to field equipment.

Ultimately, this architecture holds significant technical and operational value by realizing a 'self-evolving data flywheel (autonomous feedback loop)' that autonomously collects field uncertainty data and retrains models during normal operations—without manual labeling or human intervention.

1. Test-Time Uncertainty Estimation and Auto-Sampling

The starting point of the autonomous operational flywheel lies in the ability to precisely isolate data that the field inspection model evaluates with ambiguity. Traditional systems merely collected samples with low prediction probability, which presented a key limitation: collecting data with inherent randomness (aleatoric noise) caused by sensor noise or image corruption. To overcome this, the proposed framework leverages Monte Carlo Dropout (MC Dropout) to mathematically decompose and extract uncertainty, as illustrated in Figure 3^[6].

Figure 3. Uncertainty Decomposition Based on MC Dropout

By enabling Dropout during inference and executing multiple forward passes on the same input image, the inspection classification model outputs slightly varying prediction probability distributions (e.g., [0.9, 0.1], [0.8, 0.2], etc.). From these outputs, two forms of entropy are computed^[7].

First, measuring the 'Entropy of Expected Prediction' quantifies Total Uncertainty, which combines model knowledge deficiency (Epistemic) and inherent data ambiguity (Aleatoric).

Second, computing the 'Expected Entropy' separately quantifies only the data-inherent ambiguity (Aleatoric Uncertainty) caused by factors such as lighting reflections or noise on the image.

Finally, by calculating Mutual Information—subtracting the 'Expected Entropy' from the 'Entropy of Expected Prediction'—the data-inherent ambiguity is cancelled out, allowing precise isolation of purely 'Epistemic Uncertainty driven by model knowledge deficiency'^[8].

Applying this precise uncertainty sampling mechanism yielded an accuracy of 93.13%, representing a significant +4.84%p improvement compared to random sampling (88.29%) under identical data budget conditions. This demonstrates that focusing collection on high-value uncertainty samples maximizes training efficiency, sharply boosting model generalization performance with minimal data updates.

2. VLM-based Data Labeling

The uncertainty samples precisely selected in the previous stage bypass human operators and automatically receive ground truth labels via a Vision-Language Model (VLM). While conventional auto-labeling systems were limited to numeric outputs such as simple bounding boxes or pixel masks, VLM-based auto-labeling sets itself apart by mimicking a skilled inspector—observing product appearance and logically inferring the root causes of defects. The VLM comprehensively analyzes visual contextual differences between normal reference images and defective samples, simultaneously generating exact defect region coordinates alongside natural language explanations of defect causes and conditions, thereby dramatically enhancing engineer interpretability^[9].

To move beyond simple visual data input/output and achieve 'reasoning-driven auto-labeling', the VLM within the framework acquires domain expertise through a multi-stage training pipeline:

Stage 0: Base Model Training

Leverages industrial benchmark datasets such as Real-IAD to establish foundational cognitive capabilities for vision inspection tasks^[10]. The model learns to analyze image patches on a grid-area basis to localize defect patches, outputting concise comparative logic regarding visual differences from normal products in a standardized format.

Stage 1: Domain-Specific Fine-Tuning

Executes fine-tuning by injecting inspection data and domain knowledge from actual manufacturing sites. The model trains on paired prompt labels comprising natural language explanations that contrast subtle differences between collected [OK] and [NG] samples alongside precise defect regions, maximizing labeling accuracy for site-specific vision inspection^[11].

Stage 2: Reasoning Reinforcement Learning

Applies Group Relative Policy Optimization (GRPO)-based reinforcement learning and designs a tailored reward model to further solidify the logical consistency and reliability of inspection decisions^[12]. By defining rewards based on evaluation criteria such as "Did the model maintain a consistent reasoning process up to the final answer?" and "Did it accurately predict the ground truth location and defect cause?", the VLM is trained to provide clear labeling justification without swaying even in complex edge cases.

Completing this three-stage training pipeline allows the VLM to dramatically reduce the manual effort required for labeling while eliminating variability caused by human subjectivity, autonomously accumulating a precise, consistent, and high-quality database (Figure 4).

Figure 4. VLM Stage-by-Stage Training Pipeline (Stage 0-2)

3. Learning with Noisy Labels

While introducing VLM-based auto-labeling dramatically boosts labeling productivity and contextual reasoning capabilities, there remains a possibility that subtle label noise (noisy labels) may be introduced into pseudo-labels under extreme edge cases or optical limitation conditions. Feeding incorrectly assigned labels directly into retraining can cause the model to fall into confirmation bias—reinforcing its own errors—and lead to severe performance degradation^[13]. To prevent this, the proposed framework adopts a noise-robust learning policy based on similarity analysis within the visual feature embedding space.

The core principle lies in verifying structural consistency across the inherent distribution of visual features.

• Embedding Space & Pair Similarity

When an input image passes through the encoder, it is transformed into a high-dimensional visual feature embedding. By measuring distances to adjacent samples in the embedding space (voting by referring to the closest sample), samples that exhibit nearly identical visual features to neighboring samples yet carry conflicting labels are clearly identified as noisy label samples. For instance, this includes cases misidentified as defective (NG) despite being virtually identical in feature space to surrounding clean samples labeled as normal (OK)^[14].

• Group-wise Training Policy

Immunity to label noise is secured by applying differentiated learning loss functions and regularization policies according to the identified data groups:

Clean Set: Highly reliable samples optimized precisely using standard Cross-Entropy Loss.
Noisy Set: To offset potential label errors, Mix-up techniques—blending Clean and Noisy samples at specified ratios—and Weighted Cross-Entropy Loss are applied. Linearly interpolating visual features and labels provides regularization, effectively preventing the model from memorizing incorrect labels and causing overfitting ^[15].

Through this multi-layered noise control mechanism, performance degradation during retraining is effectively suppressed, even if some mislabeled instances contaminate the automatically collected and labeled dataset.

4. Autonomous Orchestration & Deployment

Simple rule-based automation scripts carry inherent limitations: when edge cases arise, the entire pipeline grinds to a halt or forces manual intervention by an engineer. As the core of the proposed framework, the LLM-based AI Agent acts as a central control tower governing the end-to-end autonomous operational framework, leveraging its capacities for Reasoning, Planning, Tool Calling, and Reflection on execution results^[16].

• Agent-Based Orchestration & Human Co-Piloting

The AI Agent defines each stage of the autonomous operational pipeline as modular execution tools to manage them systematically. From uncertainty data extraction and VLM-based auto-labeling to triggering GPU cluster training via the noise-robust learning pipeline and conducting simulation performance verification on the completed model, the agent monitors operations in real time and autonomously orchestrates tool execution^[4]. If exceptional events occur during training—such as convergence failure or underperformance—the agent autonomously executes replanning by adjusting hyperparameters or rebalancing data sampling ratios^[17].

Furthermore, the agent framework supports a natural language communication interface alongside its fully autonomous mode. Engineers can utilize the interactive interface to instruct the agent to reprioritize specific defect types or adjust labeling criteria and retraining thresholds in real time. By precisely interpreting user intent and immediately applying it to the workflow, the agent provides an interactive co-piloting environment that minimizes human intervention while seamlessly incorporating high-level human intuition when necessary.

• Feedback Loop-Based Autonomous Deployment

While the AI Agent possesses the capability to deploy retrained models autonomously once superior performance over the active baseline is proven, the framework incorporates a final human approval step to verify deployment safety before updating the field inspection equipment with the latest model (Continuous Deployment). Even after deployment, the framework continuously monitors the inference results and uncertainty metrics arising on the actual production line, feeding them back into the uncertainty-data collection stage to complete a self-evolving data flywheel^[18]. If post-deployment performance metrics fall below predefined thresholds, the agent automatically rolls back to the immediately preceding operational model and records the entire process in an audit log.

In this way, the Agentic AI-based autonomous operational framework shortens the lead time for responding to uncertainty on the manufacturing floor while dramatically reducing maintenance overhead.

Conclusion

Vision inspection in smart factories is evolving beyond simple 'defect detection' into a 'Data Autonomy' paradigm, where success is measured by how efficiently shop-floor data can be refined and integrated into models. Paradoxically, the human intervention and manual labeling judgment required at critical stages of conventional operational pipelines have persisted as the primary bottleneck to system scalability.

The framework presented in this article overcomes these structural limitations by combining Vision-Language Models (VLMs) with Agentic AI. From precise uncertainty sample filtering via Monte Carlo Dropout to VLM-driven contextual auto-labeling and noise-robust, group-wise training policies, tasks previously reliant on human intuition are transitioned into the domain of AI autonomy.

Furthermore, Agentic AI-driven Tool Orchestration systematically interconnects modular tools to complete a zero-human model retraining and Continuous Deployment pipeline. By offering an interactive co-piloting environment where engineers can direct system execution through natural language as needed, usability on the shop floor is maximized.

A self-evolving autonomous vision inspection framework—one that diagnoses uncertainty, infers root causes, and continuously adapts—marks a new milestone in manufacturing quality control. Minimizing human intervention while empowering AI with autonomous decision-making will serve as the core engine driving the paradigm shift toward fully autonomous operations across manufacturing sectors.

▶ VLM & Agentic AI-Based Autonomous Vision Inspection 'EXAONE Omni-Inspect' Demo

참고

[1] Gama, J., et al. (2014). "A survey on concept drift adaptation." ACM Computing Surveys (CSUR).

[2] Settles, B. (2009). "Active Learning Literature Survey." University of Wisconsin-Madison Computer Sciences Technical Report.

[3] Sambasivan, N., et al. (2021). "“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI." ACM CHI.

[4] Wu, Q., et al. (2023). "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation." arXiv preprint arXiv:2308.08155.

[5] Kim, J., et al. (2024). "VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Model." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6] Gal, Y., & Ghahramani, Z. (2016). "Dropout as a bayesian approximation: Representing model uncertainty in deep learning." International Conference on Machine Learning (ICML).

[7] Kendall, A., & Gal, Y. (2017). "What uncertainties do we need in bayesian deep learning for computer vision?" Advances in Neural Information Processing Systems (NeurIPS).

[8] Houlsby, N., et al. (2011). "Bayesian active learning for classification and preference learning." arXiv preprint arXiv:1112.5745.

[9] Liu, H., et al. (2023). "Visual Instruction Tuning." Advances in Neural Information Processing Systems (NeurIPS).

[10] Wang, C., et al. (2024). "Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11] Ouyang, L., et al. (2022). "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems (NeurIPS).

[12] Shao, Z., et al. (2024). "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv preprint arXiv:2402.03300.

[13] Song, H., et al. (2022). "Learning from noisy labels with deep neural networks: A survey." IEEE Transactions on Neural Networks and Learning Systems (TNNLS).

[14] Han, B., et al. (2018). "Co-teaching: Robust training of deep neural networks with extremely noisy labels." Advances in Neural Information Processing Systems (NeurIPS).

[15] Zhang, H., et al. (2018). "mixup: Beyond Empirical Risk Minimization." International Conference on Learning Representations (ICLR).

[16] Yao, S., et al. (2023). "ReAct: Synergizing Reasoning and Acting in Language Models." International Conference on Learning Representations (ICLR).

[17] Shinn, N., et al. (2023). "Reflexion: Language Agents with Verbal Reinforcement Learning." Advances in Neural Information Processing Systems (NeurIPS).

[18] Kreuzberger, D., et al. (2023). "Machine learning operations (mlops): Overview, definition, and architecture." IEEE Access.

목록보기