GPT-Live and GPT-Realtime: The Architectural Trade-offs Behind Continuous Voice AI

Key findings

• Conversational fluency and execution reliability are different outcomes. A frontend can sound helpful while failing to delegate the operation the customer requested.
• Control is available, but not free. Independent routing, preventive audio checks, task-state reconciliation, and verified playback introduce additional latency and operating complexity.
• First audio is not the same as useful information. Compare TTFA with time to first substantive information (TTFSI) and verified task completion.
• Architecture should follow the business risk. Evaluate representative workflows, corrections, fallback, and total cost rather than treating full-duplex conversation as proof of a better business outcome.

Introduction

In September 2026, a developer testing GPT-Live found that 14 out of 75 appointment-cancellation requests were not delegated to the backend within six seconds. Five were never delegated.

The agent had explicit instructions to delegate cancellation requests and never confirm success without backend verification. Nevertheless, it sometimes assured callers that an appointment was being canceled without actually initiating the operation.

The developer published a reproducible test after encountering the same behavior in production. In that case, the customer was told their consultation was canceled. It wasn't.

What's interesting is that GPT-Live's main architectural innovation is separating the conversational model from deeper reasoning and execution. A relatively lightweight voice model continuously handles the interaction, while another model or application performs the actual work.

But the frontend decides when the backend should participate.

So the less capable model controls access to the more capable one. If it fails to delegate, the backend's intelligence, tools, and business logic become irrelevant.

I think this illustrates a broader problem with the latest direction of voice AI. We've made impressive progress in conversational naturalness, but some of that progress comes from giving probabilistic models more authority over decisions previously controlled by more fault-tolerant and accurate logic.

And unlike an open-weight model, GPT-Live's frontend cannot be independently fine-tuned when those decisions prove unreliable.

That matters for businesses deploying AI into sales, customer service, collections, reservations, or other processes where an agent's statements can change what customers do.

There's a related problem with latency.

In enterprise discussions, I frequently hear that response latency drives conversion. That's plausible, although the evidence is less conclusive than vendor presentations often suggest.

The industry largely measures server-side time-to-first-audio (TTFA). I suspect this metric became dominant partly because it's straightforward to instrument. It tells us when the agent starts speaking, regardless of whether it has useful information, has invoked the backend, or is ready to execute the customer's request.

GPT-Live makes this distinction more consequential because speaking and reasoning are no longer tightly coupled.

The relevant business measurements should include time to first substantive information and verified task completion, not just the onset of speech.

These are not arguments against continuous voice models. They are reasons to examine the architectural trade-offs before concluding that a more natural conversation necessarily produces a better business agent.

1. What GPT-Live changes—and what it gives up

Conventional voice agents separate speech recognition, language-model reasoning, and speech synthesis.

The architecture is often criticized for accumulating latency across three sequential stages. That's accurate for naive implementations, but it overlooks how mature streaming systems work.

ASR produces partial transcripts while the caller is speaking. Turn prediction can anticipate the end of an utterance. Reasoning can begin on sufficiently stable context, and TTS can synthesize audio before the full response has been generated.

With careful orchestration, the stages overlap substantially.

Building this well is difficult. Speculative inference can waste computation, incorrect turn prediction can create conversation-breaking interruptions, and buffering between components can introduce awkward delays. But conventional pipelines are not inherently restricted to waiting for every component to finish.

They also retain several advantages that are easy to underestimate.

The application can inspect recognized speech, independently select its reasoning models, apply deterministic, ML-driven and LLM-based checks before synthesis, and control precisely what information reaches the customer.

A company can replace one component without replacing the entire system. Specialized languages, custom voices, on-premises deployment, and different inference hardware can be supported independently.

GPT-Realtime removes the mandatory external ASR and TTS stages.

It processes audio directly, performs reasoning and tool selection, and generates speech. This preserves acoustic information that transcription can discard: hesitation, emphasis, timing, and overlapping speech.

GPT-Realtime already supports streaming, interruptions, and asynchronous tool calls. Characterizing it as a simple half-duplex system would be misleading.

Its architecture is nevertheless organized primarily around model responses and their lifecycles. Speech generation, reasoning, and tool decisions are coordinated within the realtime session.

GPT-Live takes a different approach.

The voice model runs continuously, listening and speaking simultaneously. It decides when to pause, yield, backchannel, interrupt, respond directly, or delegate to a reasoning backend.

The backend may be an OpenAI Responses model or an application-owned service.

The motivation is reasonable. We don't need an expensive frontier model to decide whether the customer has finished a sentence. And a slow external service shouldn't prevent the agent from listening while work continues.

But there is a less obvious trade-off.

In a more explicitly orchestrated pipeline, the application can decide when an answer is ready to be spoken. In GPT-Live, the conversational model controls more of that decision.

The architecture improves the independence of speech from reasoning. It also reduces the application's direct authority over how the conversation progresses.

Exhibit 1. The architectural shift

These distinctions don't imply universal rankings for latency, intelligence, or reliability. Those depend on particular implementations and workloads.

The important change is who controls the boundaries between understanding, acting, and speaking.

A substantial serving-engineering achievement

OpenAI's August 2026 engineering report describes significant infrastructure work required to make GPT-Live practical.

Continuous inference is different from serving independent LLM requests.

The model must process audio on a predictable schedule while accumulating conversational state. Inference scheduling jitter becomes audible. Context compaction can invalidate the KV cache, and replacing a running instance can introduce an interruption.

OpenAI moved its media frontend and inference control implementation from Python asyncio to Go, reporting that the new system's p95 frame-delivery latency matched the previous implementation's p50.

It also introduced warm migration between stateful inference instances. A replacement instance is prepared and prefilled with compacted context while the original continues operating. The conversation moves to the replacement when it's ready.

Transport improvements, including WARP, reduce connection-establishment overhead.

This is serious engineering, and it addresses several obstacles to operating continuous speech systems commercially.

But persistent inference introduces different infrastructure economics: session affinity, memory pressure, context migration, resource allocation across concurrent calls, and potentially overlapping inference during migration.

OpenAI hasn't published enough information to compare the complete GPU economics with well-optimized alternatives under equivalent workloads.

More importantly, these optimizations do not eliminate backend latency. The infrastructure allows the frontend to continue conversing while the backend processes a request.

Whether that improves the user's experience depends on what the frontend actually says and whether the backend receives the correct task.

The research direction predates GPT-Live

Separating a fast conversational component from a slower reasoner isn't a new idea.

The 2024 Talker–Reasoner architecture distinguished immediate conversational behavior from more deliberative reasoning. Kyutai's Moshi demonstrated native full-duplex speech, while MoshiRAG explored asynchronous retrieval during conversation.

DuplexOmni, published in June 2026, explicitly separated an asynchronous interaction layer from a pluggable thinking layer.

Two more recent studies are especially relevant to the architectural trade-offs.

The University of Washington's ConvFill uses a relatively small conversational model that begins responding while incorporating knowledge from a slower reasoner. Its 18-participant user study found improved perceived responsiveness and stronger preferences on retrieval-heavy tasks.

NVIDIA's September 2026 frontend-backend architecture for tool calls trains a full-duplex speech model to emit delegation tokens and pass work to a text-based backend. It reports 92–97% tool-call recall and 81.2% accuracy in rejecting irrelevant calls in its single-turn evaluation.

These results aren't directly comparable with GPT-Live, and they don't establish production reliability.

What they show is that several teams have independently arrived at similar architectural ideas.

GPT-Live's contribution is bringing those ideas together with a sophisticated commercial serving system. Its unresolved problems are largely concentrated at the boundaries between conversational control, reasoning, and execution.

2. The frontend can fail before the backend gets involved

Evidence scope and reproduction context

The cancellation and overlapping-readback counts below are the developer's reported results in LiveKit issue #7551, not tests independently run by AgentUnicorn. This article discusses the GPT-Live 1 API and the documented model ID gpt-live-1 as of 3 November 2026. That model ID is not proof of the exact serving revision used in the September report.

The counts should remain attached to that report's setup, rather than presented as a general failure rate. A replication should retain the original prompt, audio fixtures, delegation mode, backend configuration, model ID or available snapshot, LiveKit version, timing window, and event traces. The exact serving revision and equivalence to the current deployment have not been independently verified here. Repeated trials should separately record delayed delegation, no delegation by a stated observation cutoff, backend execution, and the final appointment state.

The appointment-cancellation incident exposes a particular weakness of model-initiated delegation.

The backend cannot execute a request that never reaches it.

The developer's test does not establish a platform-wide failure rate, and future versions of GPT-Live may perform substantially better. But the mechanism is important.

The frontend decides whether to delegate. If it responds directly when it should invoke the backend, the application may receive no corresponding execution request.

Backend monitoring alone won't detect the omission. From the backend's perspective, nothing failed because nothing was attempted.

This problem becomes expensive when the frontend communicates more certainty than the underlying system can justify.

A hotel agent may confirm a cancellation that never occurred. A collections agent may acknowledge a permitted payment arrangement without updating the servicing system. An outbound sales agent may agree to a callback without recording it in the CRM.

The consequence isn't determined by the conversational error itself. It depends on whether the customer relies on the statement and what the business must do to correct it.

Overlapping speech exposes another boundary

The LiveKit report also references a related confirmation test.

When a caller answered after the agent finished reading back information, confirmation succeeded in 20 of 20 tests. When the caller answered during the readback, it succeeded in only 3 of 15.

This is particularly interesting because overlapping speech is one of the capabilities full-duplex models are intended to improve.

The conversational model may detect the interruption and respond naturally without correctly turning the customer's acknowledgment into a backend action.

There is an important distinction between hearing the user and deciding what their speech requires the system to do.

The former can improve through better audio understanding. The latter is a tool-routing and execution problem.

Changes of intent create competing versions of the task

Consider an airline agent searching for an alternative flight.

While the backend is searching, the customer changes the arrival-time requirement. The frontend acknowledges the correction, but the search may already be running under the old constraints.

If the backend returns a result based on the original instruction, which version of the task should the agent use?

What if the earlier request wasn't a search but an actual ticket modification?

Conversational interruption doesn't cancel an external operation.

OpenAI's delegation documentation explicitly assigns cancellation, task revisions, stale-result handling, and execution verification to the application.

GPT-Live maintains speculative and authoritative conversation records because live audio doesn't arrive as perfectly finalized user instructions.

That creates a trade-off between acting early and acting on stable information.

A frontend that delegates aggressively can start work sooner but may send incomplete context. A frontend that waits for certainty risks delaying useful execution.

The backend's responsibilities are also different depending on delegation mode.

With managed Responses delegation, GPT-Live prepares the request and its conversational context. With client delegation, the application builds its own backend context using transcript events and task state.

The session.delegation.created event contains metadata rather than the complete executable request.

The application must therefore reconstruct what the customer meant from an evolving transcript, where short fragments may later be corrected.

This is manageable engineering, but it weakens the assumption that a natively conversational model necessarily reduces orchestration complexity.

Execution and communication are separate outcomes

A backend operation can succeed while the customer never hears its result.

Suppose an agent creates a reservation and starts reading the confirmation. The customer interrupts before hearing the date or booking conditions.

The reservation exists, but the customer may not know that.

If they repeat the request, the application must recognize the completed transaction rather than creating another one.

The reverse problem occurs when the model communicates success before execution.

An enterprise system therefore needs to distinguish at least the intended operation, the submitted operation, the verified business-state change, and the confirmation actually delivered to the customer.

This is not unique to GPT-Live. But continuous speech creates more opportunities for these events to overlap and diverge.

Some failures are model-quality defects that could be reduced by better training. Others are distributed-systems responsibilities that remain even with a nearly perfect conversational model.

I would avoid treating them as one category.

3. The closed frontend makes model-level failures harder to correct

There's a significant difference between discovering a production weakness in an open-weight model and discovering it in GPT-Live.

With open weights, a team can collect failure examples, train targeted LoRA adapters, experiment with post-training, or modify the model's behavior for its particular workflows.

The results aren't guaranteed, and training a native speech model can be considerably harder than adapting a text LLM. But the option exists.

The current GPT-Live-1 model specification explicitly lists fine-tuning as unsupported.

Developers can tune prompts, adjust session settings, select different backend models, and build surrounding orchestration. They cannot independently change the conversational frontend's weights.

That's an important enterprise constraint.

If the backend makes an incorrect decision, an application using its own model may be able to improve that component directly.

If GPT-Live repeatedly fails to delegate a particular class of request, the developer cannot simply fine-tune the frontend on the missed examples.

The company must find a workaround within the exposed control surface or wait for OpenAI to improve the model.

This makes the scope and effectiveness of those controls especially important.

GPT-Live has guardrail mechanisms, but what do they guarantee?

The API supports server-side monitoring, sideband connections, transcript events, client delegation, instruction updates, and application-controlled audio playback.

So it would be incorrect to say that GPT-Live has no external controls.

But the controls operate at different points in the interaction, and they don't offer equivalent guarantees.

A transcript monitor can detect a problematic answer and append a corrective instruction. It cannot retract speech that has already reached the customer.

An application can block an unauthorized tool execution. That doesn't prevent the frontend from verbally claiming that the action happened.

A backend can return verified facts. The frontend can still paraphrase those facts incorrectly or fail to communicate them.

The distinction is particularly visible in OpenAI's server-side controls documentation.

OpenAI recommends monitoring transcripts while the model continues speaking. When a guardrail triggers, the application can block affected actions and inject corrective instructions.

For prevention rather than correction, however, the application needs to control audio playback.

It must buffer generated audio, collect the corresponding transcript, evaluate the content, and release approved segments—or discard them.

That is a different operating model from allowing GPT-Live to stream speech directly to the caller. And it significantly increases the cost for correct implementation of GPT-Live’s business-correct service.

The cost of preventive speech guardrails

Consider a customer asking for an account balance, an insurance coverage decision, or a confirmed booking status.

If the voice model can produce speech before that content is validated, a monitoring-only guardrail may detect an unsupported claim after the customer hears it.

A corrective message doesn't undo the original statement.

For stronger guarantees, the system needs to hold speech until it passes the relevant checks.

This introduces several compromises.

First, buffering adds latency. The application must decide how much speech constitutes a candidate segment, wait for enough transcript information to evaluate it, and run the check before releasing the audio.

A check on a short phrase may be insufficient if the harmful statement appears later in the answer. Waiting for a complete answer provides stronger context but can significantly delay the response.

Second, the application must manage playback state independently of model generation. Audio already generated may be queued, discarded, or interrupted.

Third, OpenAI documents a particularly awkward behavior: discarded speech remains in the model's conversational context.

The model can therefore continue the conversation with the assumption that it said something the customer never heard.

This is another synchronization problem created by the attempt to restore deterministic control.

The application can implement recovery policies, but the complexity is real.

The resulting architecture increasingly resembles an external speech-control layer around a model designed to manage speech autonomously.

I find that trade-off more consequential than the usual argument about whether a native speech model removes the need for ASR and TTS.

Exact wording is not guaranteed either

A similar limitation appears in mid-session instructions.

Developers reported inconsistent behavior when asking GPT-Live to deliver required phrases or disclosures.

In October, OpenAI Support confirmed that session.instructions.append can interrupt ongoing speech and that receiving the corresponding acknowledgment does not prove the requested words were spoken.

The session-management documentation recommends verified recordings or rendered audio clips when exact wording and predictable playback are required.

That is a viable mechanism.

But it moves control over consequential speech outside the model, and significantly reduces real value compared to highly-marketed use-cases.

For a language tutor, paraphrasing may be desirable. For an insurance disclosure or a customer authorization, it may not be acceptable.

The practical limitation isn't that exact communication is impossible. It's that GPT-Live's native conversational generation does not provide the necessary guarantee by itself.

Can the application bypass unreliable delegation?

There is an important nuance here.

OpenAI allows applications to process transcript fragments independently, start speculative work, and invoke application-owned backend workflows.

With Responses delegation, supported backend input events can also be used to initiate backend work outside the frontend's normal delegation decision.

So developers are not entirely dependent on GPT-Live deciding to delegate.

However, introducing an independent routing path creates additional responsibilities.

The application needs to determine which transcript fragments represent actionable intent, prevent duplicate execution when GPT-Live subsequently delegates the same request, and reconcile the result with the conversational model's own task context.

This can be done, but it requires another orchestration layer alongside the model's autonomous delegation mechanism.

The trade-off is therefore quite specific: GPT-Live can simplify conversational interaction while making deterministic business orchestration more elaborate.

Exhibit 2. Which GPT-Live controls are actually available?

The table isn't an argument that GPT-Live is uncontrollable. It shows where control moves when businesses require guarantees stronger than the conversational model provides.

For some applications, the additional engineering is justified. For others, the architecture's original advantages may become less compelling after those controls are added.

4. When filler starts increasing latency instead of hiding it

Another interesting example appeared in a November 1 LiveKit issue.

A developer reported that GPT-Live tool results were being held until an ongoing burst of filler speech finished playing.

The tool itself returned almost immediately. But the adapter delayed forwarding its result to the backend until the speech burst ended, followed by a quiet window.

Across seven measured tool turns, the reported median delay in the tool-and-hold stage was 1.83 seconds. The worst observed case reached 7.2 seconds.

A LiveKit contributor independently confirmed the relevant code path.

This is an integration defect in the LiveKit adapter, not evidence that GPT-Live inherently requires tool results to wait behind filler speech.

But it's a useful example of the coordination problems introduced by a continuous conversational frontend.

The architecture is intended to let backend work proceed independently of speech. Yet the integration ended up making backend progress depend on the completion of speech that contained no useful information.

A mechanism designed to improve perceived responsiveness was delaying the actual answer.

I think this deserves more attention than the usual comparison of first-audio latency.

TTFA may be rewarding the wrong behavior

The industry has invested heavily in reducing time-to-first-audio.

That is reasonable to a point. A 2025 study of conversational response delays, involving 54 participants, found that longer delays harmed perceived responsiveness and several aspects of user experience.

Natural fillers improved perceived response time, but did not fully compensate for the broader deterioration.

The experiment was not conducted in a production sales contact center, so it shouldn't be used to establish a numerical relationship between TTFA and conversion.

More broadly, research on contacting sales leads within minutes should not be confused with evidence about subsecond latency inside an active conversation.

For enterprise voice AI, I'd distinguish three measurements.

Time-to-first-audio measures when speech starts.

Time-to-first-substance measures when the customer receives information or a useful clarification that advances the task.

Verified task-completion time measures when the requested business operation actually succeeds.

I use Time-to-First-Substantive Information (TTFSI) as a descriptive term, not a claim that it is already a standardized industry metric.

A reproducible latency measurement protocol

For a comparison, use the end of the caller's task-bearing utterance in the captured client audio as the shared start event, t0. Keep the same input fixtures, playback path, backend delays, and network conditions across architectures. Document how utterance-end boundaries are annotated; an overlapping correction starts a separate task revision rather than silently resetting the original measurement.

• TTFA: elapsed time from t0 to the first assistant speech sample actually played to the caller. Record server-generation latency separately so transport, buffering, and playback are not hidden.
• TTFSI: elapsed time from t0 to the onset of the first played segment that contains a verified task-relevant fact, actionable result, or necessary clarification. Annotators should review the complete segment against a declared task-specific rubric; generic acknowledgments and unverified success claims do not qualify.
• Verified task-completion time: elapsed time from t0 to the authoritative business-state readback or execution acknowledgment proving that the requested change succeeded. Track when the caller hears that result as a separate event.

Use synchronized timestamps or report measured clock offsets. Retain source audio, played audio, transcript revisions, delegation/tool events, and business-state readbacks with a shared task ID. Report repeated-trial distributions, failures, observation cutoffs, and unresolved tasks—not only averages from successful calls. The numerical timelines in Figure 5 are explanatory examples, not results from this protocol.

The distinction matters particularly when a conversational frontend can speak independently of the backend.

The research supports the measurement problem

In April 2026, researchers from National Taiwan University and NVIDIA published Full-Duplex-Bench-v3.

The benchmark separated first spoken response, first tool invocation, and factual-answer delivery. It also measured filler frequency.

One result is especially relevant.

In the evaluated Ultravox configuration, filler appeared in 88% of scenarios. The model's average first tool invocation occurred more than two seconds after its first spoken response.

The researchers identified speaking before initiating the tool call as a contributor to slower factual-answer delivery.

This was a particular model and benchmark setup, not GPT-Live. The paper predates GPT-Live's public introduction and doesn't measure its performance.

It nevertheless establishes the broader phenomenon: earlier speech and earlier useful execution are not necessarily correlated.

The benchmark also found that self-corrections and complex multi-step tasks remained difficult across the evaluated architectures.

A system can be good at conversational timing while still mishandling the details that determine task success.

Filler accumulates across a conversation

I don't think conversational filler is inherently bad.

A short acknowledgment while a difficult operation runs can make the interaction less awkward. ConvFill's user study provides evidence that well-designed early responses can improve perceived responsiveness.

But the appropriate amount of filler depends on whether it contributes anything.

If nearly every turn begins with an acknowledgment before the agent reaches the useful part of its answer, the customer is repeatedly reminded that the system is processing rather than helping.

This can become particularly frustrating in longer business conversations and break rapport in non-support scenarios.

It also occupies and overloads the customer's attention and speaking channel. Instead of letting the caller clarify a request or provide additional information, the agent may continue generating speech intended mainly to conceal backend waiting.

An update based on verified progress is different from a generic reassurance. The former can be useful; the latter becomes repetitive when nothing has changed.

OpenAI now documents ways to suppress automatic acknowledgments during client delegation. The mechanism can cancel a pending acknowledgment before it is generated, but it doesn't prevent independent model speech or retract audio already produced.

This is another example of partial control: developers can steer the behavior, but silence itself is not guaranteed without managing playback externally.

I'd like to see substantially more public data on GPT-Live's behavior with realistic backend delays of 10–30 seconds.

Does the frontend continue producing filler? Does it recognize when silence is preferable? Can it provide useful progress without inventing it? What is the distribution of time-to-first-substantive-information?

We have much more public evidence about conversational naturalness than about those questions.

For a business buying voice automation, that seems like the wrong balance of evidence.

5. The recovery problem is broader than missed tool calls

Missed delegation is relatively easy to identify once execution logs are examined.

Unsupported answers are more difficult because they can sound completely normal.

Consider a hotel concierge asked whether a beach will be accessible during an upcoming stay.

The model may know the hotel's usual facilities but not have access to current restrictions, maintenance closures, or weather-related changes.

If the agent confidently says the beach is available and the customer books a particular room on that basis, the incorrect answer can have a direct commercial consequence.

That's why I wouldn't classify voice-agent risks mainly by industry.

The relevant question is how likely an error is to influence a customer's decision or initiate a consequential action.

A relatively minor factual error in hospitality may be more damaging than a harmless wording mistake in a banking conversation.

The problem becomes more complicated when the system does not recognize that its answer is weakly grounded.

What happens when the answer isn't reliable?

A production agent needs a recovery strategy.

It may need to retrieve another source, ask for clarification, invoke a stronger reasoning model, verify the result against a business record, or escalate to a human.

Those are not interchangeable options.

A stronger model may improve an ambiguous interpretation but cannot verify live inventory without an appropriate source. A second retrieval may resolve outdated information but not a missing customer authorization. A human escalation may be necessary because the business requires one, even when the model is capable of answering.

The important capability is selecting the appropriate recovery path before making an unsupported commitment.

GPT-Live's backend architecture can support these strategies. But the frontend still needs to recognize when additional work is required, unless the application detects the need independently.

That returns us to the delegation boundary.

If the voice model confidently answers from insufficient context instead of delegating, the availability of a stronger backend provides little protection.

Independent monitoring and routing can compensate, but they need to be designed and operated separately.

Audio processing can lose the most important information

There is another problem worth examining: the handling of interruptions, corrections, and short but consequential details.

Real customer conversations are messy.

A caller may change a date, correct an amount, negate a statement, or provide a crucial detail while the assistant is speaking.

These details can be acoustically difficult to recognize. They may appear in short fragments, overlap with generated speech, or be accompanied by background noise.

Aggressive denoising, endpointing, and transcript cleanup can improve apparent audio quality while risking the suppression of information the business actually needs.

This is a general risk in voice-system design, not an established GPT-Live-specific defect.

The business consequence depends on what gets lost.

In a collections call, a corrected payment amount matters far more than whether the transcript reads naturally. In a sales call, a customer's objection or changed purchasing condition may be the most valuable part of the interaction.

An agent that produces a clean conversation but loses the customer's correction has optimized the wrong thing.

This also raises an evaluation issue.

A final transcript may look coherent without proving that all important spoken information was preserved during inference.

Production tests need to examine the source audio, the interpretation available at the time of each decision, the actions taken, and the outcome.

OpenAI's GPT-Live evaluation guide explicitly recommends tracing requests through spoken intent, delegation, tool execution, resulting application state, and communication back to the user.

It also recommends repeated tests because voice-model behavior is nondeterministic.

That's the right direction.

But a test suite only identifies failures. It doesn't give the developer control over correcting the closed frontend's underlying behavior.

The ability to retrain, intervene, or replace the responsible component remains a separate architectural consideration.

This is also the boundary examined in our earlier research: Jev architecture, calibration, and open-source decision models considers more explicit decision layers, while decision models in production explains why representative evaluation, monitoring, and change control still matter after prototyping becomes easier. Neither article establishes that a decision model replaces a native speech frontend; they provide context for evaluating the business decisions around it.

6. The business case depends on the cost of the compromises

Enterprise buyers don't optimize the same metric.

Some focus mainly on price per minute. Others require a particular conversion rate, containment level, or quality threshold. A smaller group systematically compares business outcomes against the total cost of producing them.

In my experience, a buyer may reject an agent that misses its conversion target by one percentage point even if it is ten times cheaper.

I generally prefer evaluating business effect per dollar spent, while recognizing that some requirements are hard constraints rather than variables to trade against cost.

The architectural choice matters because the visible model price captures only part of the operating expense.

GPT-Live-1 is priced at $0.05 per minute for the conversational frontend, with backend reasoning and tool usage billed separately.

This can be attractive when the conversational layer handles substantial interaction and only occasionally invokes more expensive reasoning.

But the economics are different when a process requires backend work on nearly every turn.

The company pays for the continuously operating frontend alongside the backend models and tools. If the architecture also needs independent transcript interpretation, preventive audio guardrails, playback management, and additional fallback models, those resources need to be included.

So do operational costs arising from failed or duplicated actions, repeat calls, customer remediation, and human escalation.

The cost of the controls used to make GPT-Live suitable for a business process may be as important as the direct inference cost.

Better conversational benchmarks don't resolve the business comparison

OpenAI has reported meaningful improvements from GPT-Live.

For example, in early evaluations, language-learning company Speak observed nearly 80% fewer unwanted interruptions compared with earlier turn-based systems.

That's a relevant result for an application where the user needs time to think and speak hesitantly.

Other vendor-reported evaluations show improvements in conversational preference and task performance.

But these results don't establish that the architecture is better for transaction-heavy workflows or processes requiring strict controls.

The contribution of the voice model also needs to be separated from that of the backend. Comparing GPT-Live with a stronger reasoning model against GPT-Realtime with a different reasoning configuration doesn't isolate the advantage of continuous conversation.

Similarly, comparing a sophisticated native speech system with a naive cascade says little about the performance of an optimized alternative.

The most useful enterprise comparison would measure complete business outcomes under representative workloads, including failure and recovery paths.

Exhibit 3. What should be compared in production?

A model can perform exceptionally well on conversational quality while being unsuitable for a particular business process because of a control or reliability requirement.

That's not a contradiction. The metrics measure different capabilities.

7. The architectural decision is still open

I wouldn't consider GPT-Live the default architecture for enterprise voice automation. Its advances in full-duplex conversation, interruption handling, and asynchronous reasoning are substantial. But I'm not convinced that the benefits justify the additional complexity and loss of control in many business applications.

Consider latency, one of the main selling points of modern voice models.

I'd question whether an agent with a 300 ms TTFA that spends the next several seconds producing filler is actually better for business than one that starts speaking after 1000 ms but delivers useful information almost immediately.

The first agent wins the latency benchmark. The second may provide a considerably better customer experience.

The difference becomes more pronounced over a complete conversation. If the model repeatedly fills the gap between its initial response and the backend's answer, the customer has to listen to the same pattern on almost every turn. What initially sounds natural becomes irritating, wastes time, and can interfere with the customer's attempts to provide additional information.

This is why I'd put much more emphasis on time to first substantive information (TTFSI), measured alongside TTFA and actual task-completion time. A slightly slower initial response may be a perfectly reasonable trade for substantially faster access to useful information. I suspect the relative importance of these metrics varies considerably across business processes, but we have surprisingly little public evidence comparing them.

The same problem appears in the architecture's approach to execution.

GPT-Live gives the conversational frontend considerable autonomy despite its limited reasoning capabilities. It decides when to delegate, how to respond while the backend is working, and how to communicate the result. Those decisions are probabilistic, and developers cannot independently fine-tune the closed frontend when its behavior proves inadequate.

External controls are available, but their effectiveness comes with qualifications. Transcript monitoring can detect an incorrect statement after the customer hears it. Preventive speech guardrails require audio buffering and external playback control, increasing latency. Discarded speech may remain in the model's context, creating another synchronization problem. Independent tool routing can compensate for missed delegations, but it introduces additional orchestration and duplicate-execution risks.

Meanwhile, the application remains responsible for permissions, authoritative task state, cancellations, verification, idempotency, and recovery.

This creates a somewhat paradoxical situation. To make a continuously conversational model reliable enough for certain enterprise processes, developers may need to rebuild many of the controls that a more explicitly orchestrated architecture already provides.

A conventional cascade isn't automatically more reliable. Poor orchestration can cause equally serious failures. The difference is the available control surface. Developers can inspect intermediate representations, independently train or replace components, prevent unsupported content from reaching TTS, and enforce deterministic execution boundaries.

There is also no requirement to choose between completely autonomous GPT-Live and a traditional cascade. GPT-Live can operate alongside an independent ASR, reasoning, and execution pipeline, consuming verified results while that pipeline maintains control over business operations. This may offer an interesting compromise, although it also means operating two parallel interpretations of the conversation and paying for additional coordination.

The appropriate balance depends on what the agent is doing, not simply on the industry.

A hotel concierge discussing local attractions has different requirements from one modifying a nonrefundable reservation. But even an apparently harmless question about beach availability can influence a guest's purchasing decision. An insurance assistant explaining policy terminology and one communicating a coverage decision are not performing equivalent tasks.

The relevant distinction is the probability of an error changing the customer's decision and the resulting cost. More conversational autonomy may be acceptable when mistakes are inexpensive and reversible. It becomes harder to justify when incorrect information or execution failures have substantial consequences.

I also wouldn't expect model improvements alone to eliminate these trade-offs. Better training can reduce missed delegations, improve interruption handling, and make responses more reliable. It cannot remove the need to synchronize independently running transactions with an evolving conversation or guarantee that generated speech was actually heard.

What I find questionable is the industry's tendency to interpret advances in conversational naturalness as evidence of architectural superiority.

GPT-Live is an impressive technological achievement, particularly in continuous inference and serving infrastructure. But separating the conversational frontend from reasoning and execution hasn't eliminated the orchestration problem. In some respects, it has made the problem harder by transferring more decisions into a model that developers cannot directly control or retrain.

For enterprise deployments, I'd want to compare architectures on TTFSI, verified task completion, delegation reliability, grounding, recovery, and total operating costs—not just how quickly the first audio arrives or how convincingly the model handles interruptions.

I suspect those comparisons will produce less obvious winners than the current voice-model benchmarks suggest. And for many business processes, an architecture that is slightly less conversational but substantially more controllable may be the better engineering and economic decision.

Sources and evidence

Official documentation and vendor reports

• GPT-Live 1 model specification, supported features, and pricing
• Introducing GPT-Live 1 in the API: vendor-reported conversational evaluations.
• Continuous voice interaction with GPT-Live: serving and infrastructure engineering.
• Delegation guide: backend responsibilities and delegation modes.
• Server-side voice controls and live conversation management: monitoring, intervention, and playback controls.
• Voice-agent evaluation guide: tracing intent, execution, business state, and delivered communication.

Public incident reports and integration observations

• LiveKit #7551: reported cancellation and overlapping-confirmation trials; not an independently established platform-wide rate.
• LiveKit #8309: reported tool-result delay in an adapter integration, not an inherent GPT-Live latency requirement.
• OpenAI community feedback on GPT-Live 1: exact-wording and mid-session instruction observations.

Research papers and benchmark studies

• Talker–Reasoner architecture, Moshi, and MoshiRAG.
• DuplexOmni and ConvFill.
• NVIDIA frontend-backend tool-call architecture.
• Conversational response-delay study and Full-Duplex-Bench-v3.

The papers use different models, workloads, and evaluation methods. Their results support specific mechanisms or measurement questions; they are not interchangeable GPT-Live production benchmarks. API behavior and prices can change after the article's publication date.