Achieving 84% Response Quality Rate with a Self-Improving Deep Research AI Agent

In a Nutshell

Client’s Challenge

A leading SASE platform needed a unified AI agent solution that could accelerate onboarding, improve knowledge retrieval, and autonomously perform actions across multiple disconnected systems, and ensure support responses were consistently generated in a standardized format without hallucinations.

Our Solution

We built a Self-Improving AI system with a controllable level of autonomy – ranging from full human-in-the-loop oversight to fully automated operation (Claude and Agentic Skills  + LangGraph for orchestration + AWS Bedrock Claude models). The solution fuses all existing knowledge sources and support systems into a consistent set of atomic skills, each designed to return outputs in a standardized format. The entire platform was securely deployed in a cloud environment within a tunneled VPC.

Client’s Benefits

After six months, the agent supports 20+ complex engineering tasks, achieving an 84% success rate in evaluation by client’s support engineers which validated suitability of this agentic solution for production support workflows. The introduction of a repeatable HTML reporting format has reduced report review and ticket handoff times.

---

A Deep Dive

1. Overview

This project focused on building an AI-powered support automation solution for a large-scale, enterprise SaaS platform.

The solution was implemented using Generative AI and Large Language Models, combined with agentic workflows orchestrated via LangGraph and deployed securely using AWS Bedrock and cloud-native infrastructure.

We designed and delivered a self-improving, multi-agent AI system that assists support engineers in resolving customer tickets by automatically investigating issues across internal systems and generating ready-to-use responses. It improves the customer support process by accelerating ticket resolution and enhancing knowledge retrieval. By standardizing and Tak czy bez simplifying these workflows, it also significantly shortens employee onboarding, which currently takes up to six months.

Evaluation Methodology: We evaluate the agent using offline and online metrics.

Offline metrics measure internal reliability on known problem spaces and serve as a proxy for system optimization. They capture how consistently the agent handles ticket types it is expected to solve within familiar scenarios. To assess this, we use a pass² evaluation approach, where each ticket is processed through two independent agent runs. At each step, we evaluate:

• whether the correct action was selected
• whether the action was executed correctly

This methodology emphasizes consistency across independent executions, which is critical for production-grade AI systems.

Online metrics reflect real-world performance. They are based on feedback from support engineers evaluating the agent’s responses on active tickets. These metrics capture user satisfaction and provide a more accurate signal of generalization beyond predefined scenarios.

Key Objectives

• Reduce time-to-productivity for junior support engineers
• Improve consistency and reliability of support responses
• Enable autonomous and semi-autonomous execution of support actions
• Standardize outputs to eliminate hallucinations and improve verification

Key Outcomes (project ongoing)

• 84% of responses rated as good by support engineers (online metric)
• 82% pass² rate for action selection (planning) (offline metric)
• 91% pass² rate for action execution (offline metric)
• Early validation of a scalable, self-improving agent architecture suitable for production support workflows

2. Client

The client is a global provider of a cloud-native enterprise platform operating at the intersection of networking, security, and distributed systems. Their solution supports thousands of customers worldwide and underpins mission-critical infrastructure for large organizations.

The client operates in a highly complex technical domain, with fast-growing support demand driven by platform adoption, expanding product scope, and increasing system interdependencies. Support quality and scalability were identified as strategic differentiators.

3. Challenge

Support operations were constrained by long onboarding cycles, fragmented internal systems, and the need for responses that were consistent, verifiable, and safe for production use—limitations that existing AI tools failed to address end-to-end. 

At the same time, the technical complexity of the domain, coupled with strict reliability, security, and autonomy requirements, made it difficult to apply general-purpose LLMs and standard MLOps approaches without significant architectural innovation.

Business Challenges

• Reaching proficiency as a Tier 1 support engineer typically required ~6 months, with significant supervision
• Even experienced engineers struggled to efficiently retrieve information spread across multiple disconnected internal systems
• Support teams required responses that were:
  • Consistent and predictable
  • Structured rather than long, unverified text
  • Easy to validate against source systems
  • Safe from hallucinations

Previous internal AI initiatives addressed isolated use cases but were not suitable for end-to-end production support workflows.

Technology Challenges

From an LLM Perspective

• The domain relied heavily on highly specialized, internal knowledge unavailable to general-purpose models
• Strong risk of false positives and over-attribution due to noisy signals
• Complex internal APIs with non-intuitive schemas
• Individual issues often had many plausible root causes, requiring disciplined reasoning and planning

From an MLOps Perspective

• Secure connectivity to internal systems
• Robust session and state management across long, multi-step support flows
• Controlled autonomy to balance speed, safety, and reliability

4. Solution

Architectural Approach

1. We started with a fully autonomous agent to rapidly explore feasibility and failure modes
2. Through systematic error analysis and user testing, we progressively restricted autonomy where reliability was critical
3. The final solution is a self-improving, multi-agent AI architecture consisting of:
   • A Response Agent for structured, user-facing outputs
   • A Deep Research Agent for long-horizon reasoning and planning
   • A self-improvement loop, where a coding agent refines prompts, skills, and tests based on observed failures

The Deep Research Agent always produces outputs in a fixed, standardized structure, enabling fast review and validation.

A Self-improvement Loop 

Historical tickets provide a natural optimization signal: each customer question is paired with a support engineer’s investigation and final response, which serve as the gold standard. We use these examples to test whether the agent can reproduce the investigation and response using the available internal systems. Because the agent’s capabilities are defined through Markdown-based skills, coding agents such as Claude Code or Codex can iteratively improve them. Our optimization skill orchestrates this loop by reproducing historical investigations, identifying gaps, updating the relevant skills, and validating the results. Successful improvements are reviewed by developers, who can then create a pull request and merge the changes.

Solving the Business and Technical Problems

• Disparate knowledge sources were unified into atomic, human-readable agentic skills
• Undocumented operational knowledge was extracted from historical support tickets
• Every response includes direct references to underlying systems, enabling rapid verification
• All skills return outputs in a consistent HTML-based format, eliminating ambiguity

Technology Stack

• Agent orchestration: LangGraph
• Models: Anthropic Claude models via AWS Bedrock
• Backend: Python
• Secure deployment: Tunneled AWS VPC, isolated from public networks
• Self-improvement loop powered by local MCP servers

5. Process

Key Steps

1. Built a fully autonomous agent to quickly surface strengths and limitations
2. Performed structured error analysis on real support tickets
3. Transitioned to a guided, constrained agent architecture for production reliability
4. Conducted frequent UAT sessions with support engineers, iterating on structure, skills, and autonomy levels

Experts Involved

• Client-side Support Engineers
• AI Engineers and Agent Architects
• MLOps and Cloud Infrastructure Specialists

6. Outcomes

Quantitative Results

• 84% of response-crafting outputs rated as good
• 82% pass² rate for action selection
• 91% pass² rate for action execution
• Measurable early-agent performance within 6 weeks

Qualitative Results

• High acceptance of standardized, structured outputs
• Improved trust due to verifiable, system-linked responses
• Clear evidence that fine-grained control is essential for reliable agentic systems

Key Learnings

• Starting with full autonomy accelerates learning, even if it does not scale
• Production-grade agents require explicit constraints and structure
• Agents with code execution capabilities unlock powerful self-improvement loops
• Close, continuous collaboration with domain experts is non-negotiable

7. Summary

This project demonstrated how a self-improving, agentic AI system can meaningfully augment enterprise support operations in highly complex domains. Success was driven by:

• Deep collaboration with domain experts
• Willingness to rapidly refactor and change direction
• Early focus on reliability, structure, and verification—not just model capability

The resulting architecture establishes a strong foundation for scaling AI-assisted support while maintaining predictability, safety, and user trust.