Safety First: How Effective are Built-In LLM Guardrails?

Many organizations still operate under the assumption that “AI adoption” means “give everyone access to Copilot, Gemini, or Chat GPT.” This raises the question of how far organizations can rely on the built-in safeguards of these models to ensure responsible use.

Anthropic – a tech firm that markets their AI models as being especially ethical and safety conscious – describes some of their built-in safeguards as follows:

• Preventing child exploitation
• Discouraging self-harm
• Restricting information about chemical, biological, radiological, and nuclear weapons and high-yield explosives 
• Checking responses for factuality and political bias 
• Avoiding copyright infringement

But while these standard “ethical AI” safeguards are good and necessary, they don’t address the policies and day-to-day operational risks of specific organizations.  Also, the generic chatbots offered by the major AI model providers lack reporting or real policy enforcement capability.  

In other words, if an uncertified worker asks generic ChatGPT how to remove the maintenance access panel on an industrial scrap metal shredder, the chatbot can say “I don’t think you should do that” or even refuse to explain, but the worker can start a new chat and ask again without their supervisor ever hearing about it.

In Case of Emergency: Helping AI Agents Respond in the Moment

The most valuable thing about generative AI technology is that it can produce relevant responses to a staggering range of unexpected input.  For example, while traditional medical software can tell whether a patient’s temperature is above a certain threshold, generative AI can ask a patient “How are you feeling?” and respond appropriately to the patient’s story about how they felt a headache coming on last night and took a few ibuprofen which didn’t seem to help.

However, the ability to process a near infinite range of inputs comes with the ability to generate a near infinite range of outputs, including undesirable ones.  And that’s doubly true when “undesirable” output is contextual or organization specific: the latest ChatGPT model “knows” that it’s not safe to microwave aluminum foil, but it doesn’t know that servers at a particular restaurant chain aren’t allowed to refund orders above $20 without manager approval unless you tell it.

Fortunately there are a number of ways to keep AI responses within acceptable tolerances, including:

• Prompting: High-end AI models can handle far more complex instructions than most people realize (provided you structure them correctly) – so, if you instruct an AI agent not to discuss machine maintenance until it has walked through all five points of the safety guidelines (then refine those instructions through an extensive series of realistic test conversations) you can be reasonably certain the AI agent will give users the safety talk in the field. 
• Orchestration: Most agent-building platforms let you break complex workflows down into smaller steps, which can improve performance and adherence to instructions (in our example above the safety briefing could be its own prompt which must be completed before progressing to the machine maintenance prompt.) 
• Grounding: Providing an AI agent with policy documents as a reference can also help (though to see significant improvement you probably need to reformat policy docs into a more logical, machine-readable layout.)

When used together, these methods can greatly improve an AI agent’s own adherence to policy, but they still don’t give the organization visibility into AI interactions nor grant the AI agent much power to enforce policies on the user (beyond asking nicely or suggesting they talk to their manager.)

Dial 0 for Assistance – Escalating for Human Review

While there’s no dictionary definition of “AI agent”, one common feature of “agentic” systems is that they can use software tools like email or external databases.  And this is where AI finally gains some power beyond the conversation.

For example, our company’s AI agent platform allows agents to execute an “alert” command in the background, which will flag the conversation in the database and, in extreme cases, immediately generate an email to a designated human contact with a link to the transcript.  We also gave agents the ability to definitively end a conversation and even suspend access until the user is reinstated by a manager. 

When combined with clear instructions for when to fire off alerts, these tools give AI agents real enforcement and escalation capability.  

To give an example, here is the escalation logic for the AI agents we built to interact with social services clients:

Level 1: The user appears to be actively harming (or considering harming) themselves or others or is being actively abused right now or has shared plans to harm themselves or others in the immediate future (even if expressed in indirect / ambiguous / coded language); possibly mentioning a timeline, method or target. [The AI agent will implore user to call emergency services, end the conversation, and trigger an immediate email to a human social worker]  

– Level 2: User disclosed past suicide / self-harm attempt or harm to others or abuse by others either recently or without specifying how recently (but did not disclose specific plans to harm themselves or others in immediate future)  [This won’t end the conversation, but the agent will encourage user to seek help and ask if they feel like continuing while trigger an email to a human social worker]

– Level 3: User appears to feel generally suicidal, depressed, or angry towards others, but has not expressed intention to cause specific harm  [The agent will encourage user to seek help and ask if they feel like continuing, and the disclosure will be mentioned in regular reports]

– Level 4: User disclosed suicide / self-harm attempt or harm to others in distant past (but no specific risk / plans to repeat in immediate future)  [The agent will encourage user to seek help and the disclosure will be mentioned in regular reports] 

– Level 5: User passively referenced self-harm or violence against others (jokes, vague references) without referring to a specific incident or expressing clear intent. [The agent will encourage the user to seek help.]

Watching the Watchmen: Human vs. AI Monitoring

The most important thing to remember about AI is that it deals in probabilities rather than strict if/then logic, and thus is never guaranteed to give the same response to the same inputs 100% of the time.  Even if an agent is 99.999% reliable, it will make a mistake if you let it run long enough, as surely as an Olympic figure skater will eventually slip on the ice.

So what can organizations do about this?  The same thing they do for their human employees: proactively monitor and spot check the performance of AI agents to whatever extent the organization deems necessary and intervene should the error rate become unacceptable.

You can even delegate the task of monitoring to other AI agents.  For example, our company built an AI agent to advise back staff on green finance (i.e. writing business loans for energy efficiency and reliable energy upgrades.) Within 3 minutes of a conversation, a second AI agent summarizes the conversation and notes if the user might benefit from intervention by a human coach:

SUMMARY: The AI and user discussed methods to calculate CO2 savings from digitizing administrative processes, including reduced paper usage, energy consumption, and transportation emissions…  The user needed help with calculating emissions savings but demonstrated good critical thinking, and provided specific data points showing they had done preliminary research…  The user seems to be making good progress with AI assistance and doesn’t appear to need manager intervention.

Caught Doing Good: Flagging Positive Events

Most of the examples in this article have centered on risk, but it’s worth repeating that AI monitoring and escalation systems can also watch out for positive events.

For example, the AI agent we built to help social service agency clients find work will raise an alert if someone is having a mental health crisis – but it will also report if a user mentions they have an upcoming job interview.  Likewise, the agent for green finance will let managers know if a loan officer has an upcoming meeting with a commercial client. And the agents we’re currently developing for factory maintenance can log solutions to equipment problems in a database, in case anyone else runs into a similar issue in the future.

In other words, AI monitoring and escalation can do more than report problems: they can give management real time visibility into frontline operations for both risk mitigation and opportunity amplification.

Fire Drill: Developing Effective Escalation and Monitoring Frameworks

While no system for monitoring AI agent performance will be perfect, the worst system is no system at all. So what does an effective monitoring and escalation framework actually look like in practice?

Clear Guidance for Responsible AI Use

This is table stakes, but it’s also where most organizations stop. Publishing an “AI acceptable use policy” is necessary but not sufficient.  It’s like posting a speed limit sign without installing radar guns or hiring traffic cops. 

Agents with Real Enforcement and Escalation Capabilities

The difference between a policy document and an operational framework is enforcement capability. If your AI agent can only say “I don’t think you should do that” when someone asks it to help them circumvent a safety protocol, you don’t have a framework: you have a suggestion box. This is where organizations need enterprise platforms with persistent records, reporting infrastructure, and features to let AI to take action beyond the conversation itself – up to and including terminating conversations and suspending user access when safety thresholds are crossed. 

Appropriate Criteria and Thresholds for Escalation

The social services crisis escalation framework we showed earlier (with its five levels from “immediate danger” to “passive reference”) illustrates a critical principle: not every issue requires the same response, and flooding your alert channels with false positives is just as dangerous as missing real problems.

Effective escalation criteria require three things:

1. Specificity – “Flag concerning behavior” is useless. “Flag if user mentions specific plans to harm themselves or others, including timeline, method, or target” is actionable.
2. Proportionality – Match the response to the severity. Immediate email for active crises, batch reporting for lower-level concerns, silent logging for edge cases that might become patterns.
3. Testability – Run realistic scenarios through your system before deployment. If your AI agent is triggering alerts every time someone uses the word “kill” in “this project is killing me,” your thresholds need work.

The goal isn’t zero false positives (impossible) or zero false negatives (also impossible): it’s an acceptable error rate relative to human performance under similar conditions.

Technologies, Processes, and Policies for Monitoring

AI monitoring shouldn’t be a separate workflow that someone has to remember to check. It should be mainstreamed into regular reporting infrastructure (e.g., weekly summaries that flag patterns, dashboard metrics that track escalation frequency, integration with existing case management or HR systems.)

This is also where AI-monitoring-AI becomes practical. Having a second AI agent review conversation transcripts and generate summaries (like our green finance example: “User demonstrated good critical thinking… doesn’t appear to need manager intervention”) can dramatically reduce the human review burden while maintaining oversight.

The math question here is straightforward: Does the cost of AI monitoring (plus spot-check human review) deliver better outcomes than the alternatives? In most cases, the answer is yes—especially when you factor in consistency, scalability, and the ability to detect patterns across thousands of conversations that no human reviewer could spot.

Reasonable Assurance of Privacy

Here’s the tension: employees need to trust that AI interactions aren’t being weaponized against them, but organizations need visibility into what’s happening. Thread that needle poorly and people will simply refuse to use the system, defeating the entire purpose.

The solution isn’t “no monitoring” (irresponsible) or “monitor everything always” (creepy and counterproductive). It’s transparent, purpose-limited monitoring with clear policies about what gets reviewed, by whom, and under what circumstances.

For example: “All conversations are logged for quality assurance and safety purposes. Routine interactions are reviewed in aggregate only. Individual transcripts are reviewed when the AI flags a safety concern, compliance issue, or significant opportunity. Managers cannot browse transcripts at will.”

In some cases, achieving perfection is less important than being honest and transparent, as that will give people enough confidence to actually use the system instead of circumventing it with personal AI accounts and other “shadow IT.”

An Opportunity-Focused Mindset

This is the reframe that matters most: escalation frameworks aren’t just about catching problems—they’re about catching *opportunities*.

When your AI agent flags that a junior loan officer has a meeting scheduled with a major commercial client, that’s not surveillance—that’s giving senior staff the chance to provide coaching before a high-stakes interaction. When your career counseling AI notes that a client has an upcoming job interview, that’s creating an opportunity for human support at exactly the right moment.

Seat belts aren’t perfect, but they’re good enough that we feel comfortable driving. AI escalation frameworks should be viewed the same way: not as foolproof protection against every possible risk, but as reasonable safeguards that let people embrace AI technology without undue fear.

The question isn’t “Is this system perfect?” It’s “Is this system better than the alternative?”

Conclusion

The conversation about “AI safety” has been too focused on model behavior and not enough on system architecture. Responsible AI deployment isn’t just about training better models: it’s about building platforms that can actually enforce organizational policies, protect users, and create value beyond raw productivity gains.

This isn’t about “ethical AI” in the abstract. It’s about operational AI in the specific: agents that know when to escalate, systems that can monitor without micromanaging, and frameworks that treat AI as an active participant in organizational processes rather than a passive tool.

A core tension in AI deployment is that organizations adopt AI to reduce the burden on humans, but safety and compliance concerns create pressure to loop humans back in, seemingly defeating the whole purpose. But that tension dissolves when you stop thinking of AI as either “fully autonomous” or “fully supervised” and start thinking of it as a *teammate* with specific responsibilities and escalation protocols—just like you’d design for any other role in your organization.

The companies that figure out pragmatic escalation frameworks that can answer “When should this AI ask for help?” with the same clarity they’d answer “When should a junior employee ask for help?” will be the ones that actually deploy AI successfully at scale.

And here’s the punchline: escalation isn’t just about “stop, something’s wrong.” It’s about “hey, something important is happening that’s above my pay grade.” That’s a much more compelling value proposition because it means:

• Junior people get backup when they need it (not just when they screw up)
• Opportunities don’t fall through the cracks (because the AI is pattern-matching across thousands of conversations)
• Senior people get pulled in at the right moments (not drowning in alerts, but also not missing critical situations)

AI agents aren’t just monitoring systems or productivity tools. They’re becoming actors in organizational ecosystems, monitoring and being monitored by human stakeholders and each other, pursuing goals, flagging risks, and amplifying opportunities.

The question isn’t whether to build these systems. It’s whether to build them thoughtfully, on the proper technological foundation with proper guardrails and escalation frameworks, or to stumble into deployment with generic chatbots and hope for the best.

We’d argue the former is not only more responsible – it’s better business.