[AI Ethics Seminar 2025 EP.3] Agentic AI Threat Modeling and Guardrail Implementation Strategy

 

The Journey Toward Trustworthy AI: Introducing LG AI Research’s “AI Ethics Seminar”  Why does LG AI Research study AI ethics? Because we believe that AI ethics research is essential to developing AI that truly benefits people. Technology is ultimately created by humans, and as AI continues to advance, responsible development and use are just as important as technological innovation itself.  Throughout the entire AI lifecycle, we engage diverse members of our organization in open discussions and practices around AI ethics. At our AI Ethics Seminar, AI researchers, business developers, UI/UX designers, data scientists, and AI policy planners come together to explore ethical challenges in AI. The seminar serves as a forum for participants to deepen their understanding of the real value of AI ethics and to discuss how to apply ethical principles in their day-to-day work.  We hope discussions on AI ethics will spread beyond our organization and help build a broader ecosystem of trustworthy AI. In this post, we’ll introduce the key topics discussed at the 2025 AI Ethics Seminar as part of our ongoing series on responsible AI.

 

1. Moving beyond “smart conversational partners” and ushering in the era of “autonomous colleagues”

While Large Language Model (LLM)-based Generative AI has typically been viewed as a “smart conversational partner,” the emergence of Agentic AI marks a fundamental shift: AI has evolved from a mere responder to a “competent assistant” – and ultimately, an “autonomous colleague.” Agentic AI goes beyond simply answering questions. It autonomously sets goals, formulates plans, and actively interacts with its environment by leveraging external tools. This shift moves beyond the conventional “input–output” model and marks a transition to an autonomous agent that operates through a continuous loop of “perception, reasoning, action, and feedback”^{[1, 2]}.

However, the moment we grant AI not only the “ability to speak” but also the “authority to act,” a whole new class of security challenges emerges. Whereas security in the chatbot era emphasized “content safety,” such as preventing hate speech or information leakage, the security focus in the agent era shifts towards “action safety” and “goal integrity.” Action safety refers to ensuring that the actions an AI performs do not cause unintended harm or involve unauthorized use of privileges, while goal integrity means ensuring that the AI’s behavior stays aligned with the user’s intended objectives and cannot be altered through external manipulation or circumvention. Simply put, the core risk is no longer that an AI might say something wrong, but that it might do something wrong..

This article provides a structured analysis of major threat factors associated with the safe use of Agentic AI services and presents the latest research trends in effective guardrail implementation strategies, along with their practical implications.

2. Agentic AI Threat Modeling

2.1. The paradox of autonomy and security

“Agency–Security Paradox” refers to the amplification of new risks and vulnerabilities that occurs when the inherent unpredictability of LLMs relying on statistical inference is combined with execution privileges that allow them to control real systems^[3]. Previously, LLMs were mostly limited to generating text. So even if they got something wrong, the consequences rarely went beyond simply “providing incorrect information.” But things have changed. As Agentic AI gains the ability to take real actions—like modifying user accounts, deleting files, or changing system settings—the risks grow dramatically. A single incorrect answer or misaligned action could now escalate into an actual security incident. The greatest strength of an agent – “autonomy” – can paradoxically lead to the most critical security vulnerabilities.
 
These risks arise from the fact that Agentic AI operates in a loop of perception, reasoning, action, and feedback. Throughout this process, it accumulates long-term memory, calls external tools, and constantly interacts with its environment. Therefore, if its reasoning process is compromised or its memory is manipulated, the impact extends far beyond a simple response error, and it can directly translate into real system actions, potentially leading to widespread damage.

Agent memory and tool integration in particular have emerged as primary targets for attackers. In response, international security organizations such as the Open Web Application Security Project (OWASP) are establishing new threat models and security frameworks tailored specifically to the properties of Agentic AI, demonstrating a broader shift toward reshaping traditional AI security paradigms.

2.1. Key threat factors – detailed breakdown

The Image below shows the key threat factors at each stage of an agent’s perception–reasoning–action loop.

Figure 1. Threat model within a single agent^[4]

Stage	Key threat factors	Detailed threat factors
Perception - User input and external data collection	Indirect prompt injection (IPI)	T9: Identity spoofing and impersonation – Attempts to access the system and inject malicious prompts using a fake identity T13: Rogue agents within multi-agent systems – Internal or external malicious agents inject data disguised as legitimate inputs.
Memory - Storage/retrieval of past interactions and knowledge	Memory Poisoning	T1: Memory poisoning – Direct injection of malicious data into RAG/Vector databases T5: Cascading hallucination attacks – Incorrect model outputs are stored in memory, causing errors to persist/spread. T12: Agent communication poisoning – Tampering with the context/memory shared between agents
Reasoning - Planning and decision-making	Intent destruction, goal drift)	T6: Intent breaking and goal manipulation – Compromising the agent’s original intent during planning T7: Misaligned and deceptive behaviors – Failures in ethical judgment lead to the formulation of deceptive strategies. T10: Overwhelming human-in-the-loop (HITL) – Overloading the judgment/verification process to bypass approval procedures T14: Human attacks on multi-agent systems – Exploiting delegation logic complexity to trigger reasoning errors T15: Human manipulation – Crafting persuasive logic and launching social engineering attacks to deceive users into unsafe actions
Action - Tool invocation and execution	Tool misuse, confused deputy	T2: Tool misuse – Manipulating agents into abusing legitimate tools through deceptive commands to perform unintended actions T3: Privilege compromise – Gaining unauthorized elevated access to a system beyond intended permissions T4: Resource overload – Flooding an agent with requests, causing a denial of service T8: Repudiation and untraceability – Actions cannot be traced back as logs are deleted after execution. T11: Unexpected remote code execution (RCE) – Using a model, such as Code Interpreter, to generate and execute malicious code

Table 1. Threat model – detailed breakdown

1. 1. Indirect prompt injection (IPI)

   Indirect prompt injection (IPI) occurs when an attacker embeds a malicious prompt within external data sources – such as webpages or emails – that the agent can access, enabling remote control over the agent. This is extremely difficult to detect using traditional defense mechanisms because the malicious prompt enters the system indirectly through the agent’s own retrieval process during normal operations, rather than through user inputs. IPI exploits LLMs’ inability to clearly distinguish between untrusted data and system-level instructions – a vulnerability known as context mixing – which exposes any RAG-enabled Agentic AI system that references external information to potential threats.

1. 2. Intent breaking and goal manipulation

   Attackers can interfere with an agent's high-level goal-setting or detailed planning processes, inducing the agent to act in ways that deviate from its original purpose. This includes “goal drift,” a phenomenon where an agent's objectives gradually shift away from its original goals over an extended period of operation. Such manipulation can distort business processes, lead to unnecessary resource consumption, or, in severe cases, result in significant operational damage across the organization, including sensitive information leakage or the distortion of decision-making.

1. 3. Memory Poisoning

   Memory poisoning involves sneaking malicious data or false information into the long-term agent memory (such as vector databases) that the agent relies on for decision-making. Since agents treat their past interactions and stored knowledge with high confidence, any poisoned memory can lead to a cycle of repeated incorrect judgments. This attack is difficult to detect and progressively undermines the agent’s trustworthiness and stability, allowing damage to escalate over time.

1. 4. Tool misuse and privilege escalation

   This attack involves inducing an agent into abusing its legitimate tools, such as APIs and plugins, for malicious purposes. It exploits the “Confused Deputy” problem, where an agent possesses the privilege to act but lacks the ability to distinguish malicious intent in commands. Through such manipulation, attackers can bypass access-control systems and trigger harmful operations, including data deletion, unauthorized fund transfers, system configuration changes, and even remote code execution (RCE), using tools like Code Interpreter. These attacks can cause direct physical and systemic damage.

3. Guardrail strategies for Agentic AI: from static defense to dynamic adaptation

Security in the Agentic AI environment cannot be achieved by a single technology. It requires a multi-layered defense architecture that seamlessly combines model-based approaches, programmable rules, and real-time response mechanisms.

3.1. The evolution of LLM guardrails: combining model-based intelligence and programming rules

1. 1. Model-based guardrails (e.g., Qwen3Guard, Llama Guard 3)

   Whereas traditional security relies on fixed rule sets, such as firewall policies, AI security has been evolving towards a hybrid approach that flexibly leverages the judgment of model-based guardrails within the structured framework of programmable guardrails.

   
   

   

   Image 2. Qwen3Guard Architecture^[6]

   
   

   Model-based guardrails rely on a small LLM trained for security purposes to perform the contextual detection of harmfulness, jailbreak attempts, and policy violations within text. As of 2025, leading solutions include Llama Guard 3 and Qwen3Guard^{[5, 6]}.
   
   This approach offers the advantage of precisely detecting semantically harmful content such as violence, sexual content, hate speech, and jailbreak attempts. However, it has inherent limitations in its ability to evaluate tool-call parameters or business logic errors – cases where the text itself appears harmless, but the execution outcome could be dangerous.

1. 2. Programmable guardrails (e.g., NVIDIA NeMo Guardrails)

   To address these limitations, programmable frameworks, such as NVIDIA NeMo Guardrails, must serve as the core structural backbone^[7]. No matter how advanced a model may be, it cannot manage the entire logic of a complex system independently.

   
   

   

   Image 3. NeMo Guardrail Architecture^[7]

   
   

   NeMo defines dialogue flows explicitly using a modeling language called Colang and blocks risks at five critical control points based on guardrail models (e.g., Qwen3Guard) or predefined rules whenever necessary. Among these, retrieval rails and execution rails are particularly essential for defending against high-risk scenarios unique to Agentic AI, such as IPI and tool misuse.

NVIDIA NeMo Guardrails' core control components:

1. Input rail: The security model inspects the user’s prompt immediately before it reaches the system, blocking jailbreak attempts or harmful requests.

2. Dialog rails: Control the dialogue flow to ensure that it stays within predefined policies and scenarios (e.g., if a travel chatbot receives a political question, the model detects it and redirects with a pre-defined safe response).

3. Retrieval rails (RAG): Examine the content of retrieved external documents for signs of indirect prompt injection (IPI) and filter out any risky chunks.

4. Execution rails (tools): When the agent calls a tool, parameters are validated for safety in real time (e.g., detecting whether the request includes file deletion, account modification, or excessive computational load).

5. Output rails: The agent's final responses undergo a last safety check for hallucinations, sensitive-information exposure, and policy violations to ensure that they are safe and reliable for delivery.

3.2. Specialized guardrails for Agentic AI: dynamic defense for autonomous action

Unlike simple chatbots, Agentic AI executes complex tasks autonomously by using a range of tools. In doing so, the context an agent encounters changes constantly, and its actions directly impact real systems and assets. As such, predefined static rules or simple text filtering alone cannot adequately defend against these sophisticated threats.

This is why the security paradigm for Agentic AI must shift from static defense to dynamic defense. Dynamic defense refers to an adaptive security framework that tracks an agent’s state changes and the cascading impact of its actions in real time – adjusting defense strategies immediately in response. This requires the following three core mechanisms.

1. State Awareness: Agents develop multi-step plans and continuously update their reasoning to achieve their goals. Therefore, evaluating safety at a single dialog turn is inadequate. An agent's full workflow must be tracked and continuously validated for alignment between the initial goal and its current behavior, enabling the early detection of flawed planning, infinite loops, and goal drift.

2. Execution Validation: An agent's output is not just text but an API call that modifies the system. Therefore, a separate validation layer is required to verify that parameters are safe before execution. This involves simulating, sandboxing, and verifying whether a request contains destructive commands or fund transfers that exceed predefined thresholds.

3. External data isolation: Untrusted external data entering through web searches or email analysis must be isolated to prevent the corruption of prompt contexts. IPI attacks must be neutralized through real-time sanitization processes, such as filtering out hidden texts and removing HTML tags, to block prompt-context corruption before it occurs.

Image 4. Workflow of AGrail^[8]

To meet these requirements, academia and industry are actively exploring adaptive guardrail architectures like AGrail^[8]. AGrail proposes a defense system that evolves in real time in accordance with an agent's operational context and behavioral patterns, instead of relying on fixed rule sets:

1. Adaptive safety check: Rather than applying a fixed set of rules uniformly, this mechanism dynamically generates the necessary safety checks based on the task an agent is about to execute. For example, if a request to "send a report" is made, the system dynamically adds “recipient domain verification” and “confidentiality marking checks” to the inspection list on-the-fly.

2. Tool-augmented verification: In this approach, the guardrail itself leverages external tools to verify risk factors. If a URL contained in the body of the text appears suspicious, the guardrail previews it in a sandbox environment to determine its safety.

3. Collaborative experts: Rather than relying on the judgment of a single model, this approach employs multiple specialized small LLMs for collaboration to perform cross-validation. Multi-agent verification – where specialized agents handle prompt-injection detection, plan validation, and other safety-critical tasks – reduces false positives and blind spots, significantly increasing security reliability.

In sum, guardrails in the Agentic AI era are no longer fixed defense walls; they are evolving into “adaptive shields” that actively respond to changing circumstances. As greater autonomy is granted to agents, the guardrails monitoring them must possess a matching level of intelligence and adaptability to ensure controllable autonomy.

4. Operational governance and Human-in-the-Loop (HITL)

No matter how sophisticated technical solutions become, it is impossible to fully control every action of an AI system, which inherently exhibits unpredictability. Therefore, internal system guardrails alone are not sufficient; they must be complemented by operational governance that includes direct human intervention and oversight:

1. 1. Human-in-the-Loop (HITL) workflow design

   Irreversible actions, such as financial transactions, data deletion, or sending a contract, must always require explicit human approval. For example, any fund transfer exceeding a predefined amount, database deletion, or approval of outbound documents should mandate human-in-the-loop (HITL) procedures.
   
   A recommended implementation pattern for this is to use a workflow engine such as Temporal. When an agent reaches a critical decision point, it saves its state, enters a suspended status, and then restores the state and resumes execution only after receiving approval from a human administrator via a dashboard. This represents the most realistic strategy for maintaining agent autonomy while ensuring that ultimate decision authority remains with human operators.

1. 2. Memory isolation and forensic rollback

   To safeguard the memory that forms the basis of an agent’s decision-making, session isolation is essential. This prevents malicious data introduced by a prompt injection attack in one session from propagating to the global knowledge base. Furthermore, a forensic rollback system must be established – one that periodically generates snapshots of the agent's memory, similar to database backups. When anomalies are detected, this enables the system to be immediately restored to a pre-incident state. This forensic rollback mechanism serves as a critical safeguard for maintaining memory integrity.

1. 3. Adoption of an agent supervision platform

   Lastly, observability infrastructure is indispensable for transparently monitoring the entire process. Systems must secure detailed traces of an agent’s reasoning that lead up to tool invocation – mitigating the “black-box” problem and supporting internal audit requirements. By 2025, Agent supervision platforms are evolving beyond simple monitoring to incorporate active control capabilities, such as real-time anomaly detection, privilege revocation, and immediate agent shutdown. These capabilities enable the establishment of a robust control framework that can intervene instantly before flawed reasoning or errors translate into real system damage.

1. 4. Issue identification and management process

   In addition to technical measures, ensuring safe operations requires a governance framework that can quickly identify issues arising during service delivery and establish corresponding response policies. Identified issues are formalized into policies through collaboration with relevant departments within the organization. In the short term, they should be directly reflected in updates to guardrail or response-model policies, while in the long term, they may expand into broader improvement initiatives such as refining training data and operational processes when necessary. This procedural governance framework complements technical controls and serves as a key pillar for continuously improving actual service quality and safety.

5. Safe Agentic AI design: toward trustworthy autonomy

Agentic AI represents the evolution of artificial intelligence from a mere information provider to a “competent colleague” capable of performing substantive tasks. However, this advancement involves execution privileges and unpredictability that can lead to physical and business impacts on systems – making security no longer optional but a prerequisite. As discussed above, resolving the “Agency–Security Paradox” introduced by agent autonomy requires not a single technology but a multi-layered and complementary defense strategy. Only when guardrail technologies, operational governance, and process-based controls are seamlessly integrated and work in concert can we manage the potential risks posed by agents more stably.

Aligned with this paradigm, ChatEXAONE—the work agent provided for employees across the LG Group—strengthens service safety by integrating both guardrail strategy and a safety governance framework.

To begin with, ChatEXAONE incorporates a dedicated guardrail model designed to safeguard both users and the system itself. This model does more than simply detect harmful or inappropriate input; it is built to identify topics that require cultural sensitivity within the Korean social context, as well as questions that may conflict with LG Group’s corporate culture and ethical standards. For example, when queries arise that could generate misunderstandings about specific social groups or cultural elements, or when users request responses related to compliance- or ethics-sensitive corporate matters, ChatEXAONE detects such risks and adjusts its responses to avoid unnecessary conflict or information distortion.

Additionally, the service is supported by an operational safety governance framework that continuously monitors safety-related issues, evaluates whether responses and mitigations are feasible, and swiftly reflects necessary policy or standard updates in the guardrails. Through this structure, technical safeguards and organizational operating processes function in close coordination, enabling a continuous improvement cycle for enhanced safety and reliability.

Ultimately, the key to safe Agentic AI design lies in embedding “controllable autonomy” consistently across the entire system architecture. Technical defense  mechanisms and human-centered oversight processes must remain in balance to fully harness the potential of AI. From this perspective, LG AI Research is continuously exploring technologies and operational approaches that ensure “controllable autonomy”, while systematically advancing safety enhancements and security reinforcements required in real-world services.

 

AI Ethics Seminar 2025 Series

#1. [AI Ethics Seminar 2025 EP.1] How AI Is Changing Human Critical Thinking
#2. [2025 AI Ethics Seminar EP.2] Beyond bias, the journey to fair AI