Table of contents
Ahead of its September 10 public beta release, deepsense.ai received early access to OpenAI’s Agents API and tested it against several demanding workflows. We evaluated the new agent execution layer across clinical-development intelligence, controlled operational processes, software-release recovery, and business-artifact creation. Here is what we learned from putting it to work.
The Agents API puts the power of the Codex harness in developers’ hands. With persistent state, isolated execution and support for long-running workflows, businesses can build agents around their own expertise, connect them to the tools they use every day, and put them to work on complex tasks while OpenAI manages the runtime.
— Danny Wigg, Applied AI Engineer, OpenAI
Rather than evaluate it through isolated prompts or synthetic benchmarks, we wanted to see how it behaved when applied to more demanding workflows: work that spans multiple steps, interacts with tools and external sources, needs review and correction, or has to recover safely when something goes wrong.
Our primary evaluation used a clinical-development workflow inspired by a real pharma use case. We also tested controlled operational processes, software-release recovery, and presentation creation.
Across those evaluations, some of the most useful findings emerged around the operational layer: how work is persisted and recovered, how actions are tracked, how usage is measured, and what users need to see when an agent works over a longer period.
These details become part of application correctness when an agent moves from completing a task to participating in a business process.
TL;DR
- deepsense.ai received early access to OpenAI’s Agents API and tested it across several enterprise workflow patterns ahead of the public release.
- Our primary use case was Protocol Synopsis Intelligence, a workflow for reviewing early clinical-trial designs against external evidence.
- The agent’s engineering capabilities were strong. Our main feedback focused on recovery, safe retries, tool-call correlation, usage telemetry, and controlled deployment.
- The biggest lessons were operational. Once agents participate in multi-step business processes, execution state, action safety, cost, and human oversight become part of application correctness.
Testing OpenAI Agents API on a clinical trial intelligence workflow
Clinical-development teams make important study-design decisions months before a complete clinical trial protocol exists.
At that stage, they may have a protocol synopsis: a relatively short document describing the indication, patient population, endpoints, treatment arms, countries, and enrollment assumptions.
Even this early document raises questions with significant downstream consequences.
How does the proposed design compare with similar trials? Which decisions are still undefined? Are the eligibility criteria unusually restrictive? Is recruitment realistic in countries already crowded with competing studies?
Protocol Synopsis Intelligence turns the synopsis into a structured, evidence-backed review.
It extracts key design information and gaps, researches comparable studies through ClinicalTrials.gov, and produces a cited report covering:
- completeness of the protocol synopsis;
- study design and endpoint benchmarks;
- enrolment and country-level feasibility;
- population and eligibility risks;
- the competitive trial landscape.
The workflow began as a tailored demonstration inspired by a real pharma engagement. It was later generalized into an interactive demo where users can explore prepared clinical-trial examples or analyze their own protocol synopsis.
Its purpose is not to make final clinical decisions. It gives experts an earlier structured review, surfaces gaps, and turns external evidence into concrete questions for further investigation.
For our Agents API evaluation, this workflow provided a realistic product environment with domain-specific requirements, external data sources, and an existing application architecture.
We used Agents API sessions to implement, independently review, remediate, and validate changes in the active repository—testing how persistent agent work supports a broader product development lifecycle.
How we tested OpenAI Agents API with the OpenAI Agents SDK
Protocol Synopsis Intelligence was originally built with Pydantic AI to maintain a flexible, provider-agnostic foundation.
During early access, we used Agents API sessions while migrating the agentic registry-research stage to the OpenAI Agents SDK, with native MCP integration for the ClinicalTrials.gov workflow.
The agentic backbone of the resulting implementation uses OpenAI’s Agents SDK. Agents API was used during development and evaluation, including implementation, independent review, remediation, and real-browser validation in an active repository.
This allowed us to introduce and validate a new capability without rebuilding the product around it.
One useful pattern was separating work into stages.
An implementation could be completed and then independently reviewed. Problems found during review could be corrected. The result could then be validated again against the actual application.
That process uncovered both correctness and usability issues and provided a practical test of how persistent agent work behaves when a task is not finished after the first successful model response.
More broadly, Agents API can also serve as the execution backbone for agent-enabled products. Managed sessions and environments can take on more of the runtime lifecycle, with hosted or self-hosted execution depending on an organization’s requirements. Teams can retain more direct application-level control with the Agents SDK, delegate more execution infrastructure to Agents API, or combine both approaches within a broader solution.
OpenAI Agents API use cases we tested
Protocol Synopsis Intelligence was our main use case, but we deliberately tested different kinds of work.
The goal was to see which observations were specific to a research workflow and which appeared across agentic applications more broadly.
Evidence-backed research and domain intelligence
The clinical-development workflow required the agent to combine a source document with external evidence while keeping the result inspectable by domain experts.
This is an important pattern for enterprise AI because usefulness depends on more than producing a plausible answer. Users need to understand where conclusions came from, distinguish evidence from assumptions, and challenge the output when necessary.
Similar requirements appear in areas such as research, due diligence, compliance analysis, and risk screening.
Controlled agent actions and operational recovery
We also tested workflows where correctness included the actions taken during execution.
One evaluation involved synthetic insurance claims. Another focused on recovery from a faulty software release.
The agents had to investigate the situation, operate within defined constraints, respect approval points, and behave safely after an interruption.
These scenarios highlight questions that become critical in production:
If execution stops after an external action, can the system determine whether that action completed?
If a session resumes, which steps can safely run again?
Can an operator distinguish between work that failed, work that is still running, and work that completed even though the surrounding session encountered a problem?
Those questions become important quickly once agents can modify systems rather than only generate information.
AI agents for business artifact creation
Another evaluation covered presentation creation from a brief, source materials, and brand requirements.
The agent had to work iteratively rather than generate a single response: interpret source material, produce an artifact, review it against requirements, and correct the result.
This provided another test of persistent work where quality depends on a sequence of actions and evaluations rather than one model output.
What worked well in our OpenAI Agents API tests
Across our tests, the agent’s engineering capabilities were strong.
Agents API was particularly useful when work benefited from being treated as a persistent session rather than a one-off interaction.
That distinction becomes meaningful when a task includes several stages, produces intermediate artifacts, interacts with tools, or requires another agent or human to review what happened before work continues.
In the clinical-development evaluation, for example, implementation, review, correction, and validation could form parts of the same broader engineering process.
Persistent execution also creates a more natural model for asynchronous work.
Many enterprise workflows do not finish within one request. Research takes time. An operational process may wait for approval. External systems can fail temporarily. Users may need to leave and return later.
Treating that work as something with its own lifecycle is a better fit for these processes than assuming every interaction begins and ends with a synchronous application request.
Working closely with OpenAI during early access
Throughout early access, we worked closely with OpenAI, sharing practical feedback from testing in an active product repository. Our discussions focused mainly on session recovery and diagnostics, self-hosted deployment, and operational visibility, including usage and cost attribution. We saw meaningful progress across these areas as OpenAI continued to iterate on the Agents API. By the end of the early-access period, most of our findings had either been addressed or had seen substantial progress.
Four lessons for production AI agents
Our early-access work reinforced several implementation lessons that become easy to miss when evaluating agents through demos.
- Agent quality goes beyond the final answer
For multi-step agentic systems, output quality is only one dimension of correctness.
A workflow can produce a good answer and still fail operationally because an action was duplicated, an interruption left the system in an unknown state, or a user cannot determine whether the task actually completed.
Evaluation therefore needs to cover the lifecycle of the work, not just the final model response.
- Agent recovery needs to be designed upfront
It is difficult to add reliable recovery after an application has already been built around optimistic assumptions.
Teams should decide early what state needs to persist, which operations are idempotent, how interrupted work resumes, and when a human needs to make the decision.
These choices become part of the workflow architecture.
- Measure AI agent cost per business outcome
Agents make token-level comparisons increasingly incomplete.
If one workflow requires several model calls but reliably completes a high-value process, while another is cheaper per call but requires repeated human correction, token cost alone says little about their relative economics.
The useful denominator is the outcome the organization is buying: a completed investigation, reviewed document, resolved incident, processed case, or another measurable unit of work.
- Human-in-the-loop needs an operating model
“Human in the loop” is too broad to be an implementation strategy.
Teams need to define exactly when people intervene, what information they receive, which decisions require approval, and what happens when nobody responds.
Persistent execution makes those patterns easier to support, but the business still needs to define them.
Scaling from one AI agent to an enterprise portfolio
The individual workflows were different, but they exposed many of the same operational requirements: session lifecycle, state, recovery, supervision, usage tracking, and observability.
This is where economics becomes interesting. Building that foundation for one agentic application may be difficult to justify. Reusing it across multiple workflows changes the calculation.
For enterprise teams, the question becomes less about whether they can build an agent and more about whether they can establish an operating model that lets multiple agentic workflows reach production without rebuilding the same foundation each time.
What OpenAI Agents API means for enterprise AI implementation
Our early-access evaluation of OpenAI’s Agents API was encouraging.
The agent’s engineering capabilities were strong across the workflows we tested. More importantly, the evaluation helped us identify where the next layer of enterprise requirements sits: reliable recovery, safe action semantics, complete usage information, controlled deployment, and visibility appropriate for both engineers and business users.
Those requirements may sound operational rather than novel. That is precisely why they matter.
Once agents begin working across longer-running, consequential business processes, the quality of the execution layer becomes part of the quality of the application.
For enterprise teams, the practical starting point remains a bounded workflow with measurable value. Build around the actual process, define what success means, decide where humans need control, and evaluate the complete lifecycle rather than the model response alone.
For deepsense.ai, early access gave us the opportunity to test OpenAI’s Agents API against those requirements before its public release — and to provide feedback based on the implementation realities we encounter when moving AI systems toward production.
As an OpenAI implementation partner, we work with organizations from workflow discovery and architecture through implementation, integration, evaluation, and production deployment. Early access to new platform capabilities helps us understand not only what they can do, but where and how they can create value inside real enterprise systems.
Table of contents
